Jul 27 19:22:00.799155 ip-10-0-143-3 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 19:22:00.799317 ip-10-0-143-3 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 19:22:00.799586 ip-10-0-143-3 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 19:22:00.800080 ip-10-0-143-3 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 19:22:10.987318 ip-10-0-143-3 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 19:22:10.987336 ip-10-0-143-3 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ff4265e980ca4421952737e4e12007ca --
Jul 27 19:24:10.886818 ip-10-0-143-3 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 19:24:11.283035 ip-10-0-143-3 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:24:11.283035 ip-10-0-143-3 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 19:24:11.283035 ip-10-0-143-3 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:24:11.283035 ip-10-0-143-3 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 19:24:11.283035 ip-10-0-143-3 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 19:24:11.284147 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.284013    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 19:24:11.289696 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289669    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:24:11.289696 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289689    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:24:11.289696 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289695    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:24:11.289696 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289699    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:24:11.289696 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289702    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:24:11.289696 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289705    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289708    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289711    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289714    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289717    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289720    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289722    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289725    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289728    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289731    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289733    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289735    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289738    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289741    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289743    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289746    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289748    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289751    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289755    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289758    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:24:11.289935 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289761    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289764    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289767    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289770    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289773    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289775    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289778    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289780    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289783    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289785    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289788    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289790    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289793    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289795    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289799    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289802    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289804    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289807    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289809    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289812    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:24:11.290410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289815    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289818    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289820    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289823    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289825    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289828    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289830    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289832    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289835    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289838    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289841    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289843    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289846    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289848    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289850    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289854    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289856    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289858    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289861    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:24:11.290916 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289863    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289866    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289868    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289871    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289873    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289875    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289879    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289882    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289885    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289887    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289890    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289892    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289895    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289898    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289900    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289909    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289911    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289915    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289917    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289920    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:24:11.291418 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289923    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289925    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289928    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289931    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289933    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289936    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289938    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289944    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289947    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289950    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289953    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289955    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289958    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289960    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.289963    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290757    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290765    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290769    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290773    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290776    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:24:11.291924 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290779    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290781    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290784    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290787    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290789    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290792    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290794    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290797    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290799    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290802    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290805    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290807    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290810    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290813    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290816    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290818    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290821    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290823    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290825    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290828    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:24:11.292396 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290831    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290833    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290836    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290839    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290841    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290844    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290846    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290849    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290851    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290854    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290856    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290859    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290862    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290865    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290867    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290870    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290873    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290875    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290877    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:24:11.292913 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290880    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290883    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290885    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290888    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290890    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290893    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290896    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290898    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290902    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290906    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290910    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290914    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290918    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290921    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290923    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290926    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290929    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290932    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290935    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290937    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:24:11.293383 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290940    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290943    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290945    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290948    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290950    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290953    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290958    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290961    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290964    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290967    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290969    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290972    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290975    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290978    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290981    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290984    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290986    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290989    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290991    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:24:11.293876 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290994    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.290998    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291000    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291003    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291011    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291014    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291016    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291019    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291021    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291024    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291026    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291029    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291031    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291034    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291036    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291039    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291434    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291439    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291443    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291445    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:24:11.294342 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291448    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291451    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291454    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291456    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291459    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291462    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291464    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291468    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291472    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291475    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291477    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291480    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291483    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291486    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291489    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291491    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291495    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291498    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291501    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:24:11.294826 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291503    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291506    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291508    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291511    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291513    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291516    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291518    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291539    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291544    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291547    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291550    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291553    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291555    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291557    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291560    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291562    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291565    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291567    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291569    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291572    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:24:11.295295 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291574    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291576    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291579    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291581    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291584    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291586    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291590    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291593    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291597    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291601    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291604    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291606    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291609    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291611    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291614    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291616    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291618    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291621    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291623    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291627    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:24:11.295836 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291629    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291632    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291635    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291638    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291640    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291643    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291645    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291647    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291650    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291652    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291656    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291668    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291671    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291674    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291677    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291679    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291682    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291686    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291689    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:24:11.296377 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291692    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291695    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291699    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291701    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291704    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291706    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291709    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291711    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291714    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291716    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291720    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291724    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291726    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291729    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291731    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291734    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.291736    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291804    2568 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291811    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291817    2568 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291822    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 19:24:11.296869 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291826    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291829    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291833    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291837    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291840    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291843    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291847    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291850    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291853    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291856    2568 flags.go:64] FLAG: --cgroup-root=""
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291859    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291862    2568 flags.go:64] FLAG: --client-ca-file=""
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291865    2568 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291869    2568 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291874    2568 flags.go:64] FLAG: --cluster-domain=""
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291877    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291880    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291883    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291886    2568 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291889    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291892    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291895    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291898    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291901    2568 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 19:24:11.297369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291903    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291906    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291909    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291911    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291916    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291918    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291921    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291924    2568 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291927    2568 flags.go:64] FLAG: --enable-server="true"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291930    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291935    2568 flags.go:64] FLAG: --event-burst="100"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291938    2568 flags.go:64] FLAG: --event-qps="50"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291940    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291943    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291946    2568 flags.go:64] FLAG: --eviction-hard=""
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291949    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291952    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291957    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291960    2568 flags.go:64] FLAG: --eviction-soft=""
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291963    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291966    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291971    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291974    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291976    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291979    2568 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 19:24:11.297965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291982    2568 flags.go:64] FLAG: --feature-gates=""
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291986    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291988    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291991    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291994    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.291997    2568 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292000    2568 flags.go:64] FLAG: --help="false"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292002    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292005    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292008    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292011    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292014    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292017    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292020    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292022    2568 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292025    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292028    2568 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292030    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292033    2568 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292036    2568 flags.go:64] FLAG: --kube-reserved=""
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292039    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292041    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292044    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292046    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 19:24:11.298638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292050    2568 flags.go:64] FLAG: --lock-file=""
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292053    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292055    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292058    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292064    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292068    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292070    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292073    2568 flags.go:64] FLAG: --logging-format="text"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292075    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292078    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292081    2568 flags.go:64] FLAG: --manifest-url=""
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292084    2568 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292089    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292091    2568 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292095    2568 flags.go:64] FLAG: --max-pods="110"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292098    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292101    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292103    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292106    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292109    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292112    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292114    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292123    2568 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292125    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292128    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 19:24:11.299222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292131    2568 flags.go:64] FLAG: --pod-cidr=""
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292133    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292139    2568 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292142    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292145    2568 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292147    2568 flags.go:64] FLAG: --port="10250"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292150    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292153    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03a52f1f66116ce7f"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292157    2568 flags.go:64] FLAG: --qos-reserved=""
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292161    2568 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292164    2568 flags.go:64] FLAG: --register-node="true"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292167    2568 flags.go:64] FLAG: --register-with-taints=""
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292171    2568 flags.go:64] FLAG: --registry-burst="10"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292174    2568 flags.go:64] FLAG: --registry-qps="5"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292177    2568 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292180    2568 flags.go:64] FLAG: --reserved-memory=""
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292183    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292186    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292189    2568 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292192    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292194    2568 flags.go:64] FLAG: --runonce="false"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292197    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292200    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292203    2568 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292205    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 19:24:11.299896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292208    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292211    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292214    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292217    2568 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292219    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292222    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292225    2568 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292228    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292230    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292233    2568 flags.go:64] FLAG: --system-cgroups=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292235    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292241    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292243    2568 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292246    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292250    2568 flags.go:64] FLAG: --tls-min-version=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292254    2568 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292257    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292260    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292263    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292266    2568 flags.go:64] FLAG: --v="2"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292270    2568 flags.go:64] FLAG: --version="false"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292274    2568 flags.go:64] FLAG: --vmodule=""
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292278    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.292281    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 19:24:11.300502 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292402    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292406    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292409    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292412    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292415    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292417    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292419    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292422    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292424    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292427    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292430    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292432    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292435    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292437    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292439    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292442    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292444    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292447    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292449    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292452    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:24:11.301099 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292454    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292456    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292459    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292463    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292467    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292470    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292474    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292476    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292478    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292481    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292484    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292486    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292488    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292491    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292493    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292496    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292498    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292501    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292503    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:24:11.301667 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292505    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292508    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292511    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292513    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292516    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292518    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292543    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292546    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292548    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292551    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292554    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292556    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292559    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292561    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292564    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292567    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292572    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292574    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292576    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292579    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:24:11.302175 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292581    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292584    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292588    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292592    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292595    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292598    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292601    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292604    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292606    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292608    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292611    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292613    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292616    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292618    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292621    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292623    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292626    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292628    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292630    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:24:11.302677 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292633    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292637    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292640    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292643    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292646    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292648    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292651    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292653    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292656    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292659    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292662    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292664    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292667    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292670    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292672    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292674    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292677    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292680    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292683    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:24:11.303199 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292685    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.292688    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.293344    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300020    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300036    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300055    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300061    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300118    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300124    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300129    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300132    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300135    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300138    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300142    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:24:11.303687 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300144    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300147    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300149    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300152    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300155    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300157    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300160    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300162    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300165    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300167    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300170    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300173    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300175    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300178    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300180    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300183    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300185    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300188    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300191    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:24:11.304050 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300193    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300195    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300198    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300201    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300203    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300206    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300208    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300211    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300213    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300216    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300219    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300221    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300224    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300226    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300228    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300231    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300233    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300236    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300238    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300241    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:24:11.304622 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300243    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300246    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300248    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300250    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300253    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300255    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300257    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300260    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300262    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300265    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300268    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300272    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300275    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300278    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300280    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300283    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300286    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300289    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300291    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:24:11.305190 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300294    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300296    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300299    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300301    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300303    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300306    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300308    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300311    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300313    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300316    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300318    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300320    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300323    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300325    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300328    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300330    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300333    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300335    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300337    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300341    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:24:11.306114 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300344    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300347    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300349    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300351    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300354    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300357    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300359    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300362    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300365    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300367    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300370    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300372    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300375    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300377    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300382    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:24:11.306659 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300496    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300500    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300503    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300507    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300510    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300513    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300516    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300540    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300546    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300549    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300551    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300554    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300556    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300559    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300561    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300564    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300566    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300569    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300572    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300575    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 19:24:11.307262 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300577    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300580    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300582    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300585    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300588    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300590    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300593    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300595    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300597    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300600    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300602    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300605    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300607    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300610    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300613    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300617    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300619    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300623    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300626    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 19:24:11.307845 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300628    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300631    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300634    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300637    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300639    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300642    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300644    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300647    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300650    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300652    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300655    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300658    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300661    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300664    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300666    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300669    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300671    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300674    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300677    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300679    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 19:24:11.308302 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300681    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300684    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300686    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300688    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300691    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300693    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300695    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300698    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300700    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300703    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300705    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300707    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300710    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300712    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300714    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300717    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300719    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300722    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300724    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300726    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 19:24:11.308809 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300729    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300732    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300734    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300737    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300739    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300742    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300744    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300746    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300749    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300751    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300754    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300756    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300760    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300763    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300766    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300769    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300772    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300774    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300777    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 19:24:11.309307 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:11.300780    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300785    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.300891    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.302802    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.304104    2568 server.go:1019] "Starting client certificate rotation"
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.304224    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.304275    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.306251    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 19:24:11.309820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.309480    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 19:24:11.321386 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.321367    2568 log.go:25] "Validated CRI v1 runtime API"
Jul 27 19:24:11.323355 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.323335    2568 log.go:25] "Validated CRI v1 image API"
Jul 27 19:24:11.324087 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.324073    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 19:24:11.326333 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.326314    2568 fs.go:135] Filesystem UUIDs: map[17d4b2bb-7380-4338-9408-538238871d44:/dev/nvme0n1p4 4cd7806b-c585-4543-8924-6cbc23c6ee4b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 27 19:24:11.326394 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.326332    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 19:24:11.332449 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.332345    2568 manager.go:216] Machine: {Timestamp:2026-07-27 19:24:11.330989362 +0000 UTC m=+0.326837333 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104983 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2856be7d772365bf5fdc2bee45c220 SystemUUID:ec2856be-7d77-2365-bf5f-dc2bee45c220 BootID:ff4265e9-80ca-4421-9527-37e4e12007ca Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:ad:a9:52:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:ad:a9:52:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:12:52:9c:50:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 19:24:11.332449 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.332448    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 19:24:11.332572 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.332535    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 19:24:11.335485 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.335460    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 19:24:11.335668 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.335488    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 19:24:11.337281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.337268    2568 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 19:24:11.337338 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.337285    2568 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 19:24:11.337338 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.337297    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 19:24:11.338043 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.338030    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 19:24:11.338796 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.338784    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 19:24:11.339452 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.339434    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 19:24:11.340373 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340359    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 19:24:11.340516 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340504    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 19:24:11.340592 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340560    2568 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 19:24:11.340592 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340580    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 19:24:11.340689 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340600    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 19:24:11.340689 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340613    2568 kubelet.go:398] "Adding apiserver pod source"
Jul 27 19:24:11.340689 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340627    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 19:24:11.341006 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.340992    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 19:24:11.341071 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.341015    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 19:24:11.343117 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.343101    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 19:24:11.344881 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.344864    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 19:24:11.344973 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.344918    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 19:24:11.346236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346221    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346246    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346255    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346265    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346277    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346286    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346298    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 19:24:11.346310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346306    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 19:24:11.346558 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346316    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 19:24:11.346558 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346325    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 19:24:11.346558 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346364    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 19:24:11.346558 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.346374    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 19:24:11.348297 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.348284    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 19:24:11.348358 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.348299    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 19:24:11.351487 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.351470    2568 server.go:1268] "Started kubelet"
Jul 27 19:24:11.351699 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.351564    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 19:24:11.354712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.354335    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 19:24:11.354827 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.354811    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 19:24:11.354854 ip-10-0-143-3 systemd[1]: Started Kubernetes Kubelet.
Jul 27 19:24:11.355512 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.355487    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 19:24:11.355604 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.355498    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-3.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 19:24:11.355835 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.355810    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 19:24:11.357897 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.356421    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 19:24:11.358025 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.357998    2568 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 19:24:11.358642 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.358574    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 19:24:11.361316 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.360342    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-3.ec2.internal.18c63bfaf5d86b3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-3.ec2.internal,UID:ip-10-0-143-3.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-3.ec2.internal,},FirstTimestamp:2026-07-27 19:24:11.351444284 +0000 UTC m=+0.347292253,LastTimestamp:2026-07-27 19:24:11.351444284 +0000 UTC m=+0.347292253,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-3.ec2.internal,}"
Jul 27 19:24:11.363895 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.363875    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 19:24:11.363986 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.363880    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 19:24:11.364039 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.363989    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.364462 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364447    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 19:24:11.364573 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364449    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 19:24:11.364573 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364489    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 19:24:11.364680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364630    2568 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 19:24:11.364680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364649    2568 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 19:24:11.364680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364677    2568 factory.go:55] Registering systemd factory
Jul 27 19:24:11.364803 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364692    2568 factory.go:223] Registration of the systemd container factory successfully
Jul 27 19:24:11.364893 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364875    2568 factory.go:153] Registering CRI-O factory
Jul 27 19:24:11.364968 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364898    2568 factory.go:223] Registration of the crio container factory successfully
Jul 27 19:24:11.364968 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364948    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 19:24:11.364968 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364970    2568 factory.go:103] Registering Raw factory
Jul 27 19:24:11.365089 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.364981    2568 manager.go:1196] Started watching for new ooms in manager
Jul 27 19:24:11.366321 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.365973    2568 manager.go:349] Starting recovery of all containers
Jul 27 19:24:11.371948 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.371899    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 19:24:11.373625 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.373599    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmrz5"
Jul 27 19:24:11.375044 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.375017    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 19:24:11.375123 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.375098    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 19:24:11.375729 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.375715    2568 manager.go:354] Recovery completed
Jul 27 19:24:11.380128 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.380017    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.381910 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.381893    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.381988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.381926    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.381988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.381941    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.382450 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.382433    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 19:24:11.382513 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.382452    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 19:24:11.382513 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.382456    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.382513 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.382468    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 19:24:11.383101 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.383088    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.383147 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.383114    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.383147 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.383125    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.383362 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.383347    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jmrz5"
Jul 27 19:24:11.385061 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.385045    2568 policy_none.go:49] "None policy: Start"
Jul 27 19:24:11.385124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.385077    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 19:24:11.385124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.385091    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 19:24:11.386150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.386140    2568 policy_none.go:47] "Start"
Jul 27 19:24:11.417080 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417064    2568 manager.go:337] "Starting Device Plugin manager"
Jul 27 19:24:11.417158 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.417098    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 19:24:11.417158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417109    2568 server.go:87] "Starting device plugin registration server"
Jul 27 19:24:11.417312 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417297    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 19:24:11.417417 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417310    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 19:24:11.417417 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417386    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 19:24:11.417502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417461    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 19:24:11.417502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.417470    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 19:24:11.417991 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.417972    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 19:24:11.418062 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.418017    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.517792 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.517726    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.518876 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.518859    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.518945 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.518889    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.518945 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.518900    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.518945 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.518928    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.533690 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.533636    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.533690 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.533663    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-3.ec2.internal\": node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.547381 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.547355    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.561031 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.561008    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 19:24:11.561116 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.561045    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 19:24:11.561116 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.561064    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 19:24:11.561116 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.561094    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 19:24:11.562940 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.562925    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:24:11.648235 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.648193    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.661348 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.661319    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"]
Jul 27 19:24:11.661441 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.661412    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.662420 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.662404    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.662515 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.662440    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.662515 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.662454    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.663773 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.663758    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.663886 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.663871    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.663944 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.663902    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.664449 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.664431    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.664536 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.664465    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.664536 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.664484    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.664536 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.664431    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.664639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.664549    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.664639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.664581    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.665368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.665351    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dbdcd5940fe0b083a0d80c7c2740683-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal\" (UID: \"2dbdcd5940fe0b083a0d80c7c2740683\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.665404 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.665377    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dbdcd5940fe0b083a0d80c7c2740683-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal\" (UID: \"2dbdcd5940fe0b083a0d80c7c2740683\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.665440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.665398    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6180e30e1977ed4d8d23fa6e6c33c1ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-3.ec2.internal\" (UID: \"6180e30e1977ed4d8d23fa6e6c33c1ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.665872 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.665858    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.665913 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.665884    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 19:24:11.666498 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.666477    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 19:24:11.666575 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.666511    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 19:24:11.666575 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.666544    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeHasSufficientPID"
Jul 27 19:24:11.686105 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.686084    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-3.ec2.internal\" not found" node="ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.689597 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.689582    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-3.ec2.internal\" not found" node="ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.748490 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.748465    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.766499 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.766476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dbdcd5940fe0b083a0d80c7c2740683-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal\" (UID: \"2dbdcd5940fe0b083a0d80c7c2740683\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.766578 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.766502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dbdcd5940fe0b083a0d80c7c2740683-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal\" (UID: \"2dbdcd5940fe0b083a0d80c7c2740683\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.766578 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.766533    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6180e30e1977ed4d8d23fa6e6c33c1ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-3.ec2.internal\" (UID: \"6180e30e1977ed4d8d23fa6e6c33c1ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.766646 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.766583    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6180e30e1977ed4d8d23fa6e6c33c1ed-config\") pod \"kube-apiserver-proxy-ip-10-0-143-3.ec2.internal\" (UID: \"6180e30e1977ed4d8d23fa6e6c33c1ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.766646 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.766584    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dbdcd5940fe0b083a0d80c7c2740683-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal\" (UID: \"2dbdcd5940fe0b083a0d80c7c2740683\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.766646 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.766589    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dbdcd5940fe0b083a0d80c7c2740683-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal\" (UID: \"2dbdcd5940fe0b083a0d80c7c2740683\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.848886 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.848812    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.949307 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:11.949275    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:11.988849 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.988813    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:11.992888 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:11.992574    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:12.049455 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.049422    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.149975 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.149900    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.250399 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.250364    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.303931 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.303885    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 19:24:12.304353 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.304049    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 19:24:12.351422 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.351390    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.387421 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.387388    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 19:19:11 +0000 UTC" deadline="2028-04-14 21:33:01.65310524 +0000 UTC"
Jul 27 19:24:12.387421 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.387415    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15050h8m49.265693981s"
Jul 27 19:24:12.451807 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.451739    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.552213 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.552179    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.583977 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:12.583944    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbdcd5940fe0b083a0d80c7c2740683.slice/crio-908ab59bf1d842c8329cd9750b8b0ecef81964df99ff27c3dd4b2086ecbc23ac WatchSource:0}: Error finding container 908ab59bf1d842c8329cd9750b8b0ecef81964df99ff27c3dd4b2086ecbc23ac: Status 404 returned error can't find the container with id 908ab59bf1d842c8329cd9750b8b0ecef81964df99ff27c3dd4b2086ecbc23ac
Jul 27 19:24:12.588192 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.588174    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:24:12.652433 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.652405    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.652846 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:12.652819    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6180e30e1977ed4d8d23fa6e6c33c1ed.slice/crio-e675ce72fc04c95a371482fa8340c55b5f94e072c4a7429bcae4167b3e7c97fb WatchSource:0}: Error finding container e675ce72fc04c95a371482fa8340c55b5f94e072c4a7429bcae4167b3e7c97fb: Status 404 returned error can't find the container with id e675ce72fc04c95a371482fa8340c55b5f94e072c4a7429bcae4167b3e7c97fb
Jul 27 19:24:12.758483 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:12.758403    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-3.ec2.internal\" not found"
Jul 27 19:24:12.789788 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.789768    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:24:12.791238 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.791220    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:24:12.829014 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.828989    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:24:12.865093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.865068    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:12.878976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.878942    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 19:24:12.880994 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.880979    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal"
Jul 27 19:24:12.893475 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:12.893453    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 19:24:13.341979 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.341939    2568 apiserver.go:52] "Watching apiserver"
Jul 27 19:24:13.353093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.353056    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 19:24:13.353556 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.353523    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-slvtt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal","openshift-multus/multus-qd2ph","openshift-multus/network-metrics-daemon-s8jzv","openshift-network-diagnostics/network-check-target-p5xw5","openshift-ovn-kubernetes/ovnkube-node-bdqb5","kube-system/global-pull-secret-syncer-x49q7","kube-system/konnectivity-agent-qmp5w","kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal","openshift-image-registry/node-ca-dcp6x","openshift-multus/multus-additional-cni-plugins-t925l","openshift-network-operator/iptables-alerter-dkcsl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc","openshift-cluster-node-tuning-operator/tuned-pbjhs"]
Jul 27 19:24:13.356219 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.356195    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.356331 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.356293    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:13.357786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.357334    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.359748 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.359010    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.360949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.360409    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 19:24:13.360949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.360770    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n6rs8\""
Jul 27 19:24:13.361116 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.360997    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.361184 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.361148    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.361374 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.361351    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.361482 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.361426    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:13.361776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.361754    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.362053 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.362030    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 19:24:13.362146 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.362075    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzh8q\""
Jul 27 19:24:13.362329 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.362309    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 19:24:13.362791 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.362683    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:13.362791 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.362747    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:13.362945 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.362856    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.364937 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.364191    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 19:24:13.365487 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.365438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.366956 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.366935    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.369141 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.369118    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 19:24:13.369397 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.369373    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.369559 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.369516    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8hhx\""
Jul 27 19:24:13.369807 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.369789    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 19:24:13.370136 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.370092    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.370247 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.370155    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.370826 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.370807    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 19:24:13.370826 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.370818    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 19:24:13.370958 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.370873    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.370958 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.370818    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.371180 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.371161    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.371936    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.372749    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v5kjc\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.374146    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-27w7j\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.374546    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-whmfv\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.374908    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.374936    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.375116    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.375404    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.375663    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.376632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.376137    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.377156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.376889    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovnkube-script-lib\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.377156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.376934    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.377156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.376989    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7870060-1364-428b-b15d-027de7737aca-multus-daemon-config\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.377156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377067    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzvb\" (UniqueName: \"kubernetes.io/projected/b7870060-1364-428b-b15d-027de7737aca-kube-api-access-zfzvb\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.377156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377142    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-slash\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.377389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377184    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-var-lib-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.377389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377235    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.377389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377309    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.377389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377340    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-sys-fs\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.377567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377428    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdwp\" (UniqueName: \"kubernetes.io/projected/bc5cba57-c9dc-4134-9de2-990c7a424e07-kube-api-access-zmdwp\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.377567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377479    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-os-release\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.377567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377520    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:13.377733 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377580    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-systemd\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.377733 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377613    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-ovn\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.378203 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.377695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/047be226-fd26-402a-99b4-ebc0dadca574-kubelet-config\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.378298 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-registration-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.378298 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-system-cni-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.378402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378296    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-cni-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.378402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378354    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-k8s-cni-cncf-io\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.378402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378393    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtw49\" (UniqueName: \"kubernetes.io/projected/0252630f-a0ee-4786-95ab-a6cf9dec8dec-kube-api-access-rtw49\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.378556 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378441    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-node-log\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.378556 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378477    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvslg\" (UniqueName: \"kubernetes.io/projected/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-kube-api-access-pvslg\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.378556 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.378556 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378543    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-socket-dir-parent\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378574    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-conf-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378581    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-brf6w\""
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-etc-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378661    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-log-socket\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378693    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-cni-netd\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378728    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-cnibin\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.378779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-etc-kubernetes\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378785    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-run-netns\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378814    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-cni-bin\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-hostroot\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378873    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-cni-bin\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378913    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-cni-multus\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378948    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.378987    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-kubelet\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379030    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovn-node-metrics-cert\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379181 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379108    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-socket-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379211    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-device-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379231    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7870060-1364-428b-b15d-027de7737aca-cni-binary-copy\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379276    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-kubelet\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379313    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-systemd-units\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379350    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/047be226-fd26-402a-99b4-ebc0dadca574-dbus\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379419    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-netns\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-multus-certs\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379508    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.379557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379545    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovnkube-config\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.380002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379581    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-env-overrides\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.380002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379681    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 19:24:13.380002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379700    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 19:24:13.380002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379744    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 19:24:13.380002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.379908    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jcnfk\""
Jul 27 19:24:13.380357 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.380277    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.380417 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.380390    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 19:24:13.380417 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.380406    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 19:24:13.380832 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.380771    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jz9n5\""
Jul 27 19:24:13.381272 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.381251    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 19:24:13.391299 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.391273    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 19:24:13.405775 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.405647    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqxxb"
Jul 27 19:24:13.416444 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.416412    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqxxb"
Jul 27 19:24:13.465958 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.465932    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 19:24:13.480064 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480030    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-kubelet\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.480221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480079    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f592725-be92-4809-a830-e569fabe7890-agent-certs\") pod \"konnectivity-agent-qmp5w\" (UID: \"5f592725-be92-4809-a830-e569fabe7890\") " pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.480221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-device-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.480221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480134    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7870060-1364-428b-b15d-027de7737aca-cni-binary-copy\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.480221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480183    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-systemd-units\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.480364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-host\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.480364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd044254-a741-4a7e-a072-358e9b64a9f4-tmp\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.480364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480276    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-system-cni-dir\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.480364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480302    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f403b-85d1-45ee-a4b3-fdc58e281b49-host-slash\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.480364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480333    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovnkube-config\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.480364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-env-overrides\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480387    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovnkube-script-lib\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480411    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-modprobe-d\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480435    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysconfig\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480461    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480489    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480514    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-sys\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480538    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f592725-be92-4809-a830-e569fabe7890-konnectivity-ca\") pod \"konnectivity-agent-qmp5w\" (UID: \"5f592725-be92-4809-a830-e569fabe7890\") " pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.480617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480566    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-systemd\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480621    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-ovn\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-lib-modules\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cni-binary-copy\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480719    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-k8s-cni-cncf-io\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480745    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtw49\" (UniqueName: \"kubernetes.io/projected/0252630f-a0ee-4786-95ab-a6cf9dec8dec-kube-api-access-rtw49\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480786    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjp6l\" (UniqueName: \"kubernetes.io/projected/61ab69d2-8341-4561-8ed0-9873bff1eec1-kube-api-access-mjp6l\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480816    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-conf-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480853    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-etc-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480886    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-cni-netd\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480911    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysctl-d\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480936    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysctl-conf\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480962    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.481002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.480990    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481017    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-cnibin\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481042    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-etc-kubernetes\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481077    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-run-netns\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-cni-bin\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481132    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-var-lib-kubelet\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481159    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pdq\" (UniqueName: \"kubernetes.io/projected/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-kube-api-access-p4pdq\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481186    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f403b-85d1-45ee-a4b3-fdc58e281b49-iptables-alerter-script\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481222    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-hostroot\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481249    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-cni-multus\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481275    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481301    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481327    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovn-node-metrics-cert\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61ab69d2-8341-4561-8ed0-9873bff1eec1-serviceca\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-socket-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481404    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-kubelet\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481436    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-systemd\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.481577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40900963-6214-4d92-8e58-7f7ab9fff63a-tmp-dir\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481523    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98kp\" (UniqueName: \"kubernetes.io/projected/751f403b-85d1-45ee-a4b3-fdc58e281b49-kube-api-access-g98kp\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481556    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/047be226-fd26-402a-99b4-ebc0dadca574-dbus\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481582    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-netns\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481628    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-multus-certs\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481657    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481689    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2k7z\" (UniqueName: \"kubernetes.io/projected/40900963-6214-4d92-8e58-7f7ab9fff63a-kube-api-access-x2k7z\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481717    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7870060-1364-428b-b15d-027de7737aca-multus-daemon-config\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481744    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzvb\" (UniqueName: \"kubernetes.io/projected/b7870060-1364-428b-b15d-027de7737aca-kube-api-access-zfzvb\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481771    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-slash\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481799    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-var-lib-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481827    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-kubernetes\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-os-release\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481884    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481913    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481955    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-sys-fs\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481974    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdwp\" (UniqueName: \"kubernetes.io/projected/bc5cba57-c9dc-4134-9de2-990c7a424e07-kube-api-access-zmdwp\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.482327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.481992    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-os-release\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482028    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-node-log\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482049    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/047be226-fd26-402a-99b4-ebc0dadca574-kubelet-config\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482067    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-registration-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-system-cni-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482100    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-cni-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482117    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvslg\" (UniqueName: \"kubernetes.io/projected/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-kube-api-access-pvslg\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482134    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40900963-6214-4d92-8e58-7f7ab9fff63a-hosts-file\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482150    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cnibin\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482167    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482185    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-socket-dir-parent\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482201    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-log-socket\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482222    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-run\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-tuned\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28bfb\" (UniqueName: \"kubernetes.io/projected/bd044254-a741-4a7e-a072-358e9b64a9f4-kube-api-access-28bfb\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482285    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61ab69d2-8341-4561-8ed0-9873bff1eec1-host\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.483065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482312    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-cni-bin\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482385    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-cni-bin\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482420    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-kubelet\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482467    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-device-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482600    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.482671    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-systemd-units\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7870060-1364-428b-b15d-027de7737aca-cni-binary-copy\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483338    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovnkube-config\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.483694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483400    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-multus-certs\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483736    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-env-overrides\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.483724    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483772    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.483845    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:13.983803934 +0000 UTC m=+2.979651892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483861    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-socket-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.483912    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-kubelet\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484005    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/047be226-fd26-402a-99b4-ebc0dadca574-dbus\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.484127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484052    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-netns\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484232    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7870060-1364-428b-b15d-027de7737aca-multus-daemon-config\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484338    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovnkube-script-lib\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484371    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-sys-fs\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.484528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484431    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484511    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-systemd\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484522    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-slash\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484572    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-var-lib-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484588    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-run-k8s-cni-cncf-io\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484635    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-run-ovn\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-os-release\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484675    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-registration-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484694    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-conf-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484731    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-cni-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484744    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-etc-openvswitch\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484782    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-cni-netd\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484826    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484840    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/047be226-fd26-402a-99b4-ebc0dadca574-kubelet-config\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484873    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-log-socket\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484913    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc5cba57-c9dc-4134-9de2-990c7a424e07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484876    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-system-cni-dir\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.484988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.484941    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-multus-socket-dir-parent\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485008    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-run-netns\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485038    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-node-log\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485073    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-etc-kubernetes\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485086    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-host-cni-bin\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485095    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-host-var-lib-cni-multus\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485037    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-cnibin\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.485144    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:13.485196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.485169    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7870060-1364-428b-b15d-027de7737aca-hostroot\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.485479 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.485208    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:24:13.985194964 +0000 UTC m=+2.981042923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:13.489564 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.489528    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-ovn-node-metrics-cert\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.495992 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.495856    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:13.495992 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.495941    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:13.496173 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.496019    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:13.496173 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.496139    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:24:13.99612024 +0000 UTC m=+2.991968204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:13.498536 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.498511    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtw49\" (UniqueName: \"kubernetes.io/projected/0252630f-a0ee-4786-95ab-a6cf9dec8dec-kube-api-access-rtw49\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.499464 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.499414    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdwp\" (UniqueName: \"kubernetes.io/projected/bc5cba57-c9dc-4134-9de2-990c7a424e07-kube-api-access-zmdwp\") pod \"aws-ebs-csi-driver-node-jmfcc\" (UID: \"bc5cba57-c9dc-4134-9de2-990c7a424e07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.501137 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.501111    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzvb\" (UniqueName: \"kubernetes.io/projected/b7870060-1364-428b-b15d-027de7737aca-kube-api-access-zfzvb\") pod \"multus-qd2ph\" (UID: \"b7870060-1364-428b-b15d-027de7737aca\") " pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.501838 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.501810    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvslg\" (UniqueName: \"kubernetes.io/projected/41770409-4bf2-4cdd-b0e7-a84be8c44e5b-kube-api-access-pvslg\") pod \"ovnkube-node-bdqb5\" (UID: \"41770409-4bf2-4cdd-b0e7-a84be8c44e5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.567340 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.567275    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal" event={"ID":"6180e30e1977ed4d8d23fa6e6c33c1ed","Type":"ContainerStarted","Data":"e675ce72fc04c95a371482fa8340c55b5f94e072c4a7429bcae4167b3e7c97fb"}
Jul 27 19:24:13.569316 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.569283    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal" event={"ID":"2dbdcd5940fe0b083a0d80c7c2740683","Type":"ContainerStarted","Data":"908ab59bf1d842c8329cd9750b8b0ecef81964df99ff27c3dd4b2086ecbc23ac"}
Jul 27 19:24:13.582713 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582669    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-var-lib-kubelet\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.582877 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582719    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pdq\" (UniqueName: \"kubernetes.io/projected/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-kube-api-access-p4pdq\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.582877 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582772    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f403b-85d1-45ee-a4b3-fdc58e281b49-iptables-alerter-script\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.582877 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582821    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61ab69d2-8341-4561-8ed0-9873bff1eec1-serviceca\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.582877 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582843    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-var-lib-kubelet\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.583094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582899    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-systemd\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.583094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40900963-6214-4d92-8e58-7f7ab9fff63a-tmp-dir\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.583094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98kp\" (UniqueName: \"kubernetes.io/projected/751f403b-85d1-45ee-a4b3-fdc58e281b49-kube-api-access-g98kp\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.583094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.582997    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2k7z\" (UniqueName: \"kubernetes.io/projected/40900963-6214-4d92-8e58-7f7ab9fff63a-kube-api-access-x2k7z\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.583094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-kubernetes\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.583094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583063    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-os-release\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.583442 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583096    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.583442 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583158    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40900963-6214-4d92-8e58-7f7ab9fff63a-hosts-file\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.583442 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583220    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40900963-6214-4d92-8e58-7f7ab9fff63a-hosts-file\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.584009 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583980    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-kubernetes\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.584119 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.583490    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-os-release\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.584119 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584050    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61ab69d2-8341-4561-8ed0-9873bff1eec1-serviceca\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.584583 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f403b-85d1-45ee-a4b3-fdc58e281b49-iptables-alerter-script\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.584713 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584670    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cnibin\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.584785 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584724    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-run\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.584785 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584759    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-tuned\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.584893 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584768    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cnibin\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.585007 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584975    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-run\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.584786    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28bfb\" (UniqueName: \"kubernetes.io/projected/bd044254-a741-4a7e-a072-358e9b64a9f4-kube-api-access-28bfb\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585137 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585062    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61ab69d2-8341-4561-8ed0-9873bff1eec1-host\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.585295 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585272    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.585440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585423    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61ab69d2-8341-4561-8ed0-9873bff1eec1-host\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.585502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585441    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f592725-be92-4809-a830-e569fabe7890-agent-certs\") pod \"konnectivity-agent-qmp5w\" (UID: \"5f592725-be92-4809-a830-e569fabe7890\") " pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.585502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585487    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-host\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585620 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585524    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd044254-a741-4a7e-a072-358e9b64a9f4-tmp\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585620 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585551    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40900963-6214-4d92-8e58-7f7ab9fff63a-tmp-dir\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.585620 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-system-cni-dir\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.585730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585624    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-host\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-system-cni-dir\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.585730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585669    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f403b-85d1-45ee-a4b3-fdc58e281b49-host-slash\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.585730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585684    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-systemd\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585707    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-modprobe-d\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585957 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585880    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysconfig\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.585957 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-sys\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.585960    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f592725-be92-4809-a830-e569fabe7890-konnectivity-ca\") pod \"konnectivity-agent-qmp5w\" (UID: \"5f592725-be92-4809-a830-e569fabe7890\") " pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.586065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586001    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-lib-modules\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586041    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cni-binary-copy\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586085    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjp6l\" (UniqueName: \"kubernetes.io/projected/61ab69d2-8341-4561-8ed0-9873bff1eec1-kube-api-access-mjp6l\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysctl-d\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-sys\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysctl-conf\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586192    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586230    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.586278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586274    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-modprobe-d\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586651 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586348    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysconfig\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.586651 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f403b-85d1-45ee-a4b3-fdc58e281b49-host-slash\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.586891 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586868    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f592725-be92-4809-a830-e569fabe7890-konnectivity-ca\") pod \"konnectivity-agent-qmp5w\" (UID: \"5f592725-be92-4809-a830-e569fabe7890\") " pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.586965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.586911    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.587040 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.587022    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-lib-modules\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.587099 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.587027    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysctl-d\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.587196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.587181    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-sysctl-conf\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.587349 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.587332    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.590648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.587969    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-cni-binary-copy\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.590648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.588993    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bd044254-a741-4a7e-a072-358e9b64a9f4-etc-tuned\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.590648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.590589    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd044254-a741-4a7e-a072-358e9b64a9f4-tmp\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.591600 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.591522    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f592725-be92-4809-a830-e569fabe7890-agent-certs\") pod \"konnectivity-agent-qmp5w\" (UID: \"5f592725-be92-4809-a830-e569fabe7890\") " pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.596528 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.596455    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pdq\" (UniqueName: \"kubernetes.io/projected/2ae699bd-8bb1-49f1-a257-bb1f406c1d22-kube-api-access-p4pdq\") pod \"multus-additional-cni-plugins-t925l\" (UID: \"2ae699bd-8bb1-49f1-a257-bb1f406c1d22\") " pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.597250 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.597227    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2k7z\" (UniqueName: \"kubernetes.io/projected/40900963-6214-4d92-8e58-7f7ab9fff63a-kube-api-access-x2k7z\") pod \"node-resolver-slvtt\" (UID: \"40900963-6214-4d92-8e58-7f7ab9fff63a\") " pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.599143 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.599116    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28bfb\" (UniqueName: \"kubernetes.io/projected/bd044254-a741-4a7e-a072-358e9b64a9f4-kube-api-access-28bfb\") pod \"tuned-pbjhs\" (UID: \"bd044254-a741-4a7e-a072-358e9b64a9f4\") " pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.600428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.599789    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjp6l\" (UniqueName: \"kubernetes.io/projected/61ab69d2-8341-4561-8ed0-9873bff1eec1-kube-api-access-mjp6l\") pod \"node-ca-dcp6x\" (UID: \"61ab69d2-8341-4561-8ed0-9873bff1eec1\") " pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.600428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.600230    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98kp\" (UniqueName: \"kubernetes.io/projected/751f403b-85d1-45ee-a4b3-fdc58e281b49-kube-api-access-g98kp\") pod \"iptables-alerter-dkcsl\" (UID: \"751f403b-85d1-45ee-a4b3-fdc58e281b49\") " pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.673360 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.673320    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc"
Jul 27 19:24:13.683966 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:13.683882    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5cba57_c9dc_4134_9de2_990c7a424e07.slice/crio-ebc442be7a7fedd8193116cdc0b8994b17c34355fa94c81da0d7c2e6d51f16b7 WatchSource:0}: Error finding container ebc442be7a7fedd8193116cdc0b8994b17c34355fa94c81da0d7c2e6d51f16b7: Status 404 returned error can't find the container with id ebc442be7a7fedd8193116cdc0b8994b17c34355fa94c81da0d7c2e6d51f16b7
Jul 27 19:24:13.686178 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.686149    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qd2ph"
Jul 27 19:24:13.696127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.696084    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:13.703371 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.703338    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pbjhs"
Jul 27 19:24:13.711856 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.711831    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-slvtt"
Jul 27 19:24:13.719963 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.719610    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:13.729731 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.729694    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcp6x"
Jul 27 19:24:13.740821 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.740780    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dkcsl"
Jul 27 19:24:13.746748 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.746720    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t925l"
Jul 27 19:24:13.884154 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.884058    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 19:24:13.989691 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.989651    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:13.989868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:13.989721    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:13.989868 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.989815    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:13.989868 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.989859    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:13.990019 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.989881    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:24:14.989862788 +0000 UTC m=+3.985710752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:13.990019 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:13.989911    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:14.989894678 +0000 UTC m=+3.985742637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:14.090653 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:14.090616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:14.090840 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.090810    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:14.090840 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.090832    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:14.090935 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.090847    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:14.090935 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.090916    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:24:15.090894686 +0000 UTC m=+4.086742666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:14.419003 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:14.418954    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:19:13 +0000 UTC" deadline="2028-02-11 06:18:06.808104188 +0000 UTC"
Jul 27 19:24:14.419003 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:14.418993    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13522h53m52.389116033s"
Jul 27 19:24:14.572782 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:14.572744    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc" event={"ID":"bc5cba57-c9dc-4134-9de2-990c7a424e07","Type":"ContainerStarted","Data":"ebc442be7a7fedd8193116cdc0b8994b17c34355fa94c81da0d7c2e6d51f16b7"}
Jul 27 19:24:14.757545 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:14.757477    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41770409_4bf2_4cdd_b0e7_a84be8c44e5b.slice/crio-11a94774ceb5034134d51d1fc6cf25eadfc3ab4d56ab7cb9f228510e02e8a241 WatchSource:0}: Error finding container 11a94774ceb5034134d51d1fc6cf25eadfc3ab4d56ab7cb9f228510e02e8a241: Status 404 returned error can't find the container with id 11a94774ceb5034134d51d1fc6cf25eadfc3ab4d56ab7cb9f228510e02e8a241
Jul 27 19:24:14.759551 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:14.758986    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7870060_1364_428b_b15d_027de7737aca.slice/crio-14f7f20637c5bbdc5b035915db70eaec2ed75c3023f7195e787152208a4f8d9a WatchSource:0}: Error finding container 14f7f20637c5bbdc5b035915db70eaec2ed75c3023f7195e787152208a4f8d9a: Status 404 returned error can't find the container with id 14f7f20637c5bbdc5b035915db70eaec2ed75c3023f7195e787152208a4f8d9a
Jul 27 19:24:14.768484 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:14.768442    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40900963_6214_4d92_8e58_7f7ab9fff63a.slice/crio-e92205e808392ce244710c2a8c95d49444b439cb9fc19656a7e4702d2204b01c WatchSource:0}: Error finding container e92205e808392ce244710c2a8c95d49444b439cb9fc19656a7e4702d2204b01c: Status 404 returned error can't find the container with id e92205e808392ce244710c2a8c95d49444b439cb9fc19656a7e4702d2204b01c
Jul 27 19:24:14.769135 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:14.769104    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae699bd_8bb1_49f1_a257_bb1f406c1d22.slice/crio-b8ba3b641affd3e45203c30c6d697657ec3e7f9fd7e500df12a9de3ef821f500 WatchSource:0}: Error finding container b8ba3b641affd3e45203c30c6d697657ec3e7f9fd7e500df12a9de3ef821f500: Status 404 returned error can't find the container with id b8ba3b641affd3e45203c30c6d697657ec3e7f9fd7e500df12a9de3ef821f500
Jul 27 19:24:14.769389 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:14.769368    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd044254_a741_4a7e_a072_358e9b64a9f4.slice/crio-6bc75e5babba6af0554305a6a78603378ef669af388c2105a24caf583568c29a WatchSource:0}: Error finding container 6bc75e5babba6af0554305a6a78603378ef669af388c2105a24caf583568c29a: Status 404 returned error can't find the container with id 6bc75e5babba6af0554305a6a78603378ef669af388c2105a24caf583568c29a
Jul 27 19:24:14.997802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:14.997578    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:14.997955 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:14.997864    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:14.998014 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.997750    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:14.998068 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.998021    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:16.998007841 +0000 UTC m=+5.993855802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:14.998304 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.997968    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:14.998304 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:14.998282    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:24:16.998260487 +0000 UTC m=+5.994108458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:15.098978 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.098938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:15.099182 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.099099    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:15.099182 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.099120    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:15.099182 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.099134    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:15.099351 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.099205    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:24:17.099184746 +0000 UTC m=+6.095032725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:15.419763 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.419695    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:19:13 +0000 UTC" deadline="2028-02-14 01:18:53.847396727 +0000 UTC"
Jul 27 19:24:15.419763 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.419759    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13589h54m38.427642234s"
Jul 27 19:24:15.565633 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.564700    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:15.565633 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.564879    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:15.565916 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.565697    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:15.565916 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.565824    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:15.566445 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.564593    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:15.566445 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:15.566404    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:15.597401 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.597356    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcp6x" event={"ID":"61ab69d2-8341-4561-8ed0-9873bff1eec1","Type":"ContainerStarted","Data":"733664ad98b2fd8765a58235c3e240a456878641cf4278b63b54ed6c3a5de932"}
Jul 27 19:24:15.603041 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.602993    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pbjhs" event={"ID":"bd044254-a741-4a7e-a072-358e9b64a9f4","Type":"ContainerStarted","Data":"6bc75e5babba6af0554305a6a78603378ef669af388c2105a24caf583568c29a"}
Jul 27 19:24:15.608694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.608649    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerStarted","Data":"b8ba3b641affd3e45203c30c6d697657ec3e7f9fd7e500df12a9de3ef821f500"}
Jul 27 19:24:15.618856 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.618761    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-slvtt" event={"ID":"40900963-6214-4d92-8e58-7f7ab9fff63a","Type":"ContainerStarted","Data":"e92205e808392ce244710c2a8c95d49444b439cb9fc19656a7e4702d2204b01c"}
Jul 27 19:24:15.627077 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.626968    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qd2ph" event={"ID":"b7870060-1364-428b-b15d-027de7737aca","Type":"ContainerStarted","Data":"14f7f20637c5bbdc5b035915db70eaec2ed75c3023f7195e787152208a4f8d9a"}
Jul 27 19:24:15.639326 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.639197    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkcsl" event={"ID":"751f403b-85d1-45ee-a4b3-fdc58e281b49","Type":"ContainerStarted","Data":"c0ffc49f1d343e9bc4d65b2c95523ab2b1c5c2c11097b21c3feeed23ce6cb22b"}
Jul 27 19:24:15.641298 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.640976    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qmp5w" event={"ID":"5f592725-be92-4809-a830-e569fabe7890","Type":"ContainerStarted","Data":"d3d1165d4aecbd4d03d68fb8c7021b18f531a6ab0eab7a7ed41a5d8203507270"}
Jul 27 19:24:15.653800 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.653756    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"11a94774ceb5034134d51d1fc6cf25eadfc3ab4d56ab7cb9f228510e02e8a241"}
Jul 27 19:24:15.669163 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.669121    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal" event={"ID":"6180e30e1977ed4d8d23fa6e6c33c1ed","Type":"ContainerStarted","Data":"f1a42a8b32dc1329ba7f02f9b6f6f165cd1406eb88c508feb9011399cbea2624"}
Jul 27 19:24:15.692421 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.692326    2568 generic.go:356] "Generic (PLEG): container finished" podID="2dbdcd5940fe0b083a0d80c7c2740683" containerID="58e6a815174dabef86a6403b5a40e317519472982c0cba416c949699b9c6dae9" exitCode=0
Jul 27 19:24:15.692421 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.692386    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal" event={"ID":"2dbdcd5940fe0b083a0d80c7c2740683","Type":"ContainerDied","Data":"58e6a815174dabef86a6403b5a40e317519472982c0cba416c949699b9c6dae9"}
Jul 27 19:24:15.729151 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:15.729079    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-3.ec2.internal" podStartSLOduration=2.729057085 podStartE2EDuration="2.729057085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:15.697815578 +0000 UTC m=+4.693663562" watchObservedRunningTime="2026-07-27 19:24:15.729057085 +0000 UTC m=+4.724905067"
Jul 27 19:24:16.724145 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:16.724038    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal" event={"ID":"2dbdcd5940fe0b083a0d80c7c2740683","Type":"ContainerStarted","Data":"d9f58c9e1e77ec029e76983f197fe6c02d860946f4393f823d3fb5cc5ca162b2"}
Jul 27 19:24:17.016858 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:17.016053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:17.016858 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:17.016166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:17.016858 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.016271    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:17.016858 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.016322    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:24:21.016306052 +0000 UTC m=+10.012154010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:17.016858 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.016706    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:17.016858 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.016765    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:21.016748197 +0000 UTC m=+10.012596159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:17.122650 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:17.120202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:17.122650 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.120458    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:17.122650 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.120482    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:17.122650 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.120497    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:17.122650 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.120570    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:24:21.120548204 +0000 UTC m=+10.116396163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:17.561784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:17.561746    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:17.561971 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.561882    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:17.562290 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:17.562272    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:17.562708 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.562679    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:17.562872 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:17.562854    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:17.562954 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:17.562938    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:19.563039 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:19.562633    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:19.563039 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:19.562659    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:19.563039 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:19.562643    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:19.563039 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:19.562779    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:19.563039 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:19.562855    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:19.563039 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:19.562938    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:21.052422 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:21.052378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:21.052897 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:21.052438    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:21.052897 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.052572    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:21.052897 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.052584    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:21.052897 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.052660    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:24:29.052633747 +0000 UTC m=+18.048481722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:21.052897 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.052681    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:29.052670926 +0000 UTC m=+18.048518887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:21.153585 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:21.153550    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:21.154216 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.153767    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:21.154216 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.153797    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:21.154216 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.153811    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:21.154216 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.153874    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:24:29.153853818 +0000 UTC m=+18.149701778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:21.562949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:21.562499    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:21.562949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:21.562619    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:21.562949 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.562631    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:21.562949 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.562754    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:21.562949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:21.562810    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:21.562949 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:21.562872    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:23.562310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:23.561851    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:23.562310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:23.561866    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:23.562310 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:23.561866    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:23.562310 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:23.561976    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:23.562310 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:23.562041    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:23.562310 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:23.562100    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:25.562028 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:25.561986    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:25.562457 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:25.561997    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:25.562457 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:25.562123    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:25.562457 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:25.562120    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:25.562457 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:25.562197    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:25.562457 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:25.562312    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:27.561820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:27.561781    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:27.562229 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:27.561836    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:27.562229 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:27.561782    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:27.562229 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:27.561921    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:27.562229 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:27.562013    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:27.562229 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:27.562110    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:29.111582 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:29.111517    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:29.112062 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:29.111614    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:29.112062 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.111704    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:29.112062 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.111751    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:29.112062 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.111791    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:24:45.111769301 +0000 UTC m=+34.107617270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:29.112062 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.111812    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:45.111802138 +0000 UTC m=+34.107650094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:29.212573 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:29.212539    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:29.212768 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.212676    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:29.212768 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.212690    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:29.212768 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.212699    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:29.212768 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.212759    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:24:45.212745086 +0000 UTC m=+34.208593046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:29.562297 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:29.562258    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:29.562469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:29.562309    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:29.562469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:29.562267    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:29.562469 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.562384    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:29.562617 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.562464    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:29.562617 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:29.562565    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:31.562507 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:31.562466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:31.563010 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:31.562605    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:31.563010 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:31.562627    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:31.563010 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:31.562636    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:31.563010 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:31.562808    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:31.563010 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:31.562885    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:33.563672 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.563647    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:33.564084 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:33.563773    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:33.564084 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.563907    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:33.564084 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:33.564010    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:33.564245 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.564118    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:33.564245 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:33.564222    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:33.765901 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.765448    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qmp5w" event={"ID":"5f592725-be92-4809-a830-e569fabe7890","Type":"ContainerStarted","Data":"3fdca2568e94a704b8f6df63e4c519af70cfb79eec6873123e76d0aed3602d7d"}
Jul 27 19:24:33.767248 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.767221    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc" event={"ID":"bc5cba57-c9dc-4134-9de2-990c7a424e07","Type":"ContainerStarted","Data":"1cb2e159a57f96b4c483e7f79fad8d1728ba40e376a1e9d6b4319d14330cead4"}
Jul 27 19:24:33.769556 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.769246    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcp6x" event={"ID":"61ab69d2-8341-4561-8ed0-9873bff1eec1","Type":"ContainerStarted","Data":"46160372c67fb9f80a10e9933953e1a8e800355e1935799e693edf4660a83887"}
Jul 27 19:24:33.771252 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.770753    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pbjhs" event={"ID":"bd044254-a741-4a7e-a072-358e9b64a9f4","Type":"ContainerStarted","Data":"da4e6496dbd6d55f2fde6dc055659e3fb0d912088e1eb6b549af2d386669b92b"}
Jul 27 19:24:33.772714 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.772695    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerStarted","Data":"6e4aee8be508e81b5ba0d4c3cfc2919203c45f9ebf21c074a2687da6f2a83a7f"}
Jul 27 19:24:33.774287 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.774263    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-slvtt" event={"ID":"40900963-6214-4d92-8e58-7f7ab9fff63a","Type":"ContainerStarted","Data":"9f6499b2171166852ee9d8363c6663244a720234958097cdb4dab89b30ed4f9a"}
Jul 27 19:24:33.783589 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.783301    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-3.ec2.internal" podStartSLOduration=20.783286215 podStartE2EDuration="20.783286215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:16.744833647 +0000 UTC m=+5.740681629" watchObservedRunningTime="2026-07-27 19:24:33.783286215 +0000 UTC m=+22.779134193"
Jul 27 19:24:33.784187 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.783806    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qmp5w" podStartSLOduration=3.099631885 podStartE2EDuration="21.783794435s" totalImagesPullingTime="18.68416255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:33.783028588 +0000 UTC m=+22.778876566" watchObservedRunningTime="2026-07-27 19:24:33.783794435 +0000 UTC m=+22.779642413"
Jul 27 19:24:33.801413 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.801373    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dcp6x" podStartSLOduration=3.112194812 podStartE2EDuration="21.801357424s" totalImagesPullingTime="18.689162612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:33.800760418 +0000 UTC m=+22.796608428" watchObservedRunningTime="2026-07-27 19:24:33.801357424 +0000 UTC m=+22.797205475"
Jul 27 19:24:33.849737 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.849679    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-slvtt" podStartSLOduration=3.202998514 podStartE2EDuration="21.849658496s" totalImagesPullingTime="18.646659982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:33.848962643 +0000 UTC m=+22.844810634" watchObservedRunningTime="2026-07-27 19:24:33.849658496 +0000 UTC m=+22.845506475"
Jul 27 19:24:33.871808 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.871755    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pbjhs" podStartSLOduration=3.1814176610000002 podStartE2EDuration="21.87173334s" totalImagesPullingTime="18.690315679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:33.871667677 +0000 UTC m=+22.867515656" watchObservedRunningTime="2026-07-27 19:24:33.87173334 +0000 UTC m=+22.867581319"
Jul 27 19:24:33.901069 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.900906    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:33.973517 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:33.973468    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:34.134221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.134195    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:34.134984 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.134966    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:34.777614 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.777585    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qd2ph" event={"ID":"b7870060-1364-428b-b15d-027de7737aca","Type":"ContainerStarted","Data":"45cf3a61298b67fbc04bfeecba3c9d27f96071edc4709b5623c048c44165d489"}
Jul 27 19:24:34.780511 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.780488    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"4db252f0edfd8e66c174bf9de3c71d8f54373521142993328fc6856ae522cb3b"}
Jul 27 19:24:34.780601 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.780583    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"8ea599be16bcc62157ffdb3dda249f0011858e257ee3b3d7d9ee53dcdd8fb7d9"}
Jul 27 19:24:34.780642 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.780633    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"ac2c05fe56b7e474c3321be193ba5ecb460698def26a17417668fc6abab10e0c"}
Jul 27 19:24:34.780683 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.780676    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"7a4b851852f7d425282f7b48a53cfd68c59c8772a3d40b4aca8b9aee4364ceb3"}
Jul 27 19:24:34.780739 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.780727    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"c8510e097008b483a0053a11ecea330df8b2f20e7420cbbec1cd5a0ab033c779"}
Jul 27 19:24:34.780781 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.780761    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"3e9621737755d13396a8fa5ecb019fa72b4bc663c41d84e9a5ae9a88dc575fde"}
Jul 27 19:24:34.781784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.781764    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ae699bd-8bb1-49f1-a257-bb1f406c1d22" containerID="6e4aee8be508e81b5ba0d4c3cfc2919203c45f9ebf21c074a2687da6f2a83a7f" exitCode=0
Jul 27 19:24:34.781885 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.781833    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerDied","Data":"6e4aee8be508e81b5ba0d4c3cfc2919203c45f9ebf21c074a2687da6f2a83a7f"}
Jul 27 19:24:34.782294 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.782256    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:34.782989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.782942    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qmp5w"
Jul 27 19:24:34.801687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.801643    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qd2ph" podStartSLOduration=3.798925192 podStartE2EDuration="22.801629101s" totalImagesPullingTime="19.002703909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:34.801025323 +0000 UTC m=+23.796873301" watchObservedRunningTime="2026-07-27 19:24:34.801629101 +0000 UTC m=+23.797477116"
Jul 27 19:24:34.931544 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:34.931480    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:35.042056 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.041984    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:35.116154 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.116132    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 19:24:35.434199 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.434052    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T19:24:35.116148003Z","UUID":"4d9f890b-c8eb-4651-94cf-bda09297b2ed","Handler":null,"Name":"","Endpoint":""}
Jul 27 19:24:35.435494 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.435479    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 19:24:35.435583 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.435537    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 19:24:35.562435 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.561970    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:35.562435 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.561982    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:35.562435 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:35.562094    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:35.562435 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.562167    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:35.562435 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:35.562165    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:35.562435 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:35.562258    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:35.786555 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.786500    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc" event={"ID":"bc5cba57-c9dc-4134-9de2-990c7a424e07","Type":"ContainerStarted","Data":"a44641ecb56817b19f87360a7de9be2ad53f3b44791cadda4cbef3732eae48c1"}
Jul 27 19:24:35.788155 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.788126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dkcsl" event={"ID":"751f403b-85d1-45ee-a4b3-fdc58e281b49","Type":"ContainerStarted","Data":"e58438dc074315494cfdd72efd124d9d80c29544125565ecab42b10b2337dd4f"}
Jul 27 19:24:35.807460 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.807408    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dkcsl" podStartSLOduration=5.11083249 podStartE2EDuration="23.807388288s" totalImagesPullingTime="18.696555798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:35.807349583 +0000 UTC m=+24.803197561" watchObservedRunningTime="2026-07-27 19:24:35.807388288 +0000 UTC m=+24.803236248"
Jul 27 19:24:35.938281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:35.938243    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:36.042899 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:36.042825    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:36.793016 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:36.792977    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc" event={"ID":"bc5cba57-c9dc-4134-9de2-990c7a424e07","Type":"ContainerStarted","Data":"0aee55dbcaa6174df2094e0b0af75d597aecb2712f5e19be79fe0117236a30cc"}
Jul 27 19:24:36.797932 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:36.797888    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"839c34e6087be61dea6b37ac086d337938d162320506f7323130fa6006cd8696"}
Jul 27 19:24:36.815502 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:36.815447    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmfcc" podStartSLOduration=2.688382169 podStartE2EDuration="24.815401362s" totalImagesPullingTime="22.127019193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:36.814320651 +0000 UTC m=+25.810168631" watchObservedRunningTime="2026-07-27 19:24:36.815401362 +0000 UTC m=+25.811249343"
Jul 27 19:24:37.561441 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:37.561405    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:37.561692 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:37.561405    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:37.561692 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:37.561542    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:37.561692 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:37.561404    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:37.561692 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:37.561650    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:37.561692 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:37.561683    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:38.180052 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:38.180013    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:38.250591 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:38.250556    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:38.342022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:38.341989    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:39.562293 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.562151    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:39.562896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.562173    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:39.562896 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:39.562371    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:39.562896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.562200    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:39.562896 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:39.562492    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:39.562896 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:39.562604    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:39.808745 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.808711    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" event={"ID":"41770409-4bf2-4cdd-b0e7-a84be8c44e5b","Type":"ContainerStarted","Data":"e8577aed0af0bc4c130e6de787041c8274a419cdc32119eaf5cacf79207bfdb7"}
Jul 27 19:24:39.808984 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.808950    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:39.810560 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.810537    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ae699bd-8bb1-49f1-a257-bb1f406c1d22" containerID="32daf45c18c5d2a9c9b3074a33c7879714942b5279a0e060bbc7ba77203aeaa9" exitCode=0
Jul 27 19:24:39.810674 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.810577    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerDied","Data":"32daf45c18c5d2a9c9b3074a33c7879714942b5279a0e060bbc7ba77203aeaa9"}
Jul 27 19:24:39.825656 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:39.825634    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:40.506204 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:40.504268    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5" podStartSLOduration=9.546606656 podStartE2EDuration="28.504243056s" totalImagesPullingTime="18.9576364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:40.502374795 +0000 UTC m=+29.498222776" watchObservedRunningTime="2026-07-27 19:24:40.504243056 +0000 UTC m=+29.500091034"
Jul 27 19:24:40.813857 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:40.813693    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:40.814262 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:40.813921    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:40.828772 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:40.828746    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:24:41.562593 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:41.562570    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:41.562593 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:41.562593    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:41.562779 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:41.562653    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:41.562779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:41.562720    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:41.562843 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:41.562812    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:41.562897 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:41.562881    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:41.818591 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:41.818489    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ae699bd-8bb1-49f1-a257-bb1f406c1d22" containerID="b1f2b8d4373a10653d90f2bee92d4dec5e3a2093da4ee0bcf389b8ba2cf2dd71" exitCode=0
Jul 27 19:24:41.818929 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:41.818577    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerDied","Data":"b1f2b8d4373a10653d90f2bee92d4dec5e3a2093da4ee0bcf389b8ba2cf2dd71"}
Jul 27 19:24:42.045409 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:42.045385    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x49q7"]
Jul 27 19:24:42.045623 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:42.045492    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:42.045623 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:42.045597    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:42.047284 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:42.047244    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p5xw5"]
Jul 27 19:24:42.047411 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:42.047357    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:42.047477 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:42.047456    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:42.048015 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:42.047986    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s8jzv"]
Jul 27 19:24:42.048116 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:42.048102    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:42.048232 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:42.048210    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:43.561719 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:43.561690    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:43.562158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:43.561700    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:43.562158 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:43.561814    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:43.562158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:43.561700    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:43.562158 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:43.561901    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:43.562158 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:43.561991    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:43.825727 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:43.825696    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ae699bd-8bb1-49f1-a257-bb1f406c1d22" containerID="8a5b7219f6f0448b29f0ead307c7d942416393b567f43d80127e7f40c25df811" exitCode=0
Jul 27 19:24:43.825877 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:43.825740    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerDied","Data":"8a5b7219f6f0448b29f0ead307c7d942416393b567f43d80127e7f40c25df811"}
Jul 27 19:24:45.125165 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:45.125128    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:45.125624 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:45.125225    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:45.125624 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.125299    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:45.125624 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.125325    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:45.125624 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.125389    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret podName:047be226-fd26-402a-99b4-ebc0dadca574 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:17.125357889 +0000 UTC m=+66.121205845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret") pod "global-pull-secret-syncer-x49q7" (UID: "047be226-fd26-402a-99b4-ebc0dadca574") : object "kube-system"/"original-pull-secret" not registered
Jul 27 19:24:45.125624 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.125411    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:25:17.125400225 +0000 UTC m=+66.121248182 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 19:24:45.226355 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:45.226320    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:45.226545 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.226510    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 19:24:45.226618 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.226556    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 19:24:45.226618 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.226570    2568 projected.go:196] Error preparing data for projected volume kube-api-access-htztt for pod openshift-network-diagnostics/network-check-target-p5xw5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:45.226710 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.226638    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt podName:4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf nodeName:}" failed. No retries permitted until 2026-07-27 19:25:17.226616388 +0000 UTC m=+66.222464348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-htztt" (UniqueName: "kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt") pod "network-check-target-p5xw5" (UID: "4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 19:24:45.561663 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:45.561629    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:45.561846 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:45.561629    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:45.561846 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.561782    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:24:45.561846 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.561817    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p5xw5" podUID="4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf"
Jul 27 19:24:45.561846 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:45.561629    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:45.562049 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:45.561913    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x49q7" podUID="047be226-fd26-402a-99b4-ebc0dadca574"
Jul 27 19:24:47.324184 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.324021    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-3.ec2.internal" event="NodeReady"
Jul 27 19:24:47.324640 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.324298    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 19:24:47.382052 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.382013    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xrhc7"]
Jul 27 19:24:47.384466 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.384446    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kkx66"]
Jul 27 19:24:47.384618 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.384598    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.386155 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.386137    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.387259 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.387235    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 19:24:47.387409 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.387391    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 19:24:47.387481 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.387414    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 19:24:47.387481 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.387394    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ssdbw\""
Jul 27 19:24:47.388656 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.388634    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 19:24:47.388743 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.388724    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 19:24:47.388883 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.388833    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksszw\""
Jul 27 19:24:47.399828 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.399773    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrhc7"]
Jul 27 19:24:47.401669 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.401643    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kkx66"]
Jul 27 19:24:47.544142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.544103    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6852432-508f-4243-84ef-f9aebd2e2ae3-config-volume\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.544347 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.544171    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c9cba3-d810-4681-b777-a3bb5abad710-cert\") pod \"ingress-canary-xrhc7\" (UID: \"39c9cba3-d810-4681-b777-a3bb5abad710\") " pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.544347 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.544259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkzc\" (UniqueName: \"kubernetes.io/projected/a6852432-508f-4243-84ef-f9aebd2e2ae3-kube-api-access-gkkzc\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.544347 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.544315    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6852432-508f-4243-84ef-f9aebd2e2ae3-tmp-dir\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.544347 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.544339    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wft2b\" (UniqueName: \"kubernetes.io/projected/39c9cba3-d810-4681-b777-a3bb5abad710-kube-api-access-wft2b\") pod \"ingress-canary-xrhc7\" (UID: \"39c9cba3-d810-4681-b777-a3bb5abad710\") " pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.544580 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.544423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.565909 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.565883    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:24:47.566067 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.565980    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:24:47.566140 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.566076    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:24:47.569145 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.569126    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 19:24:47.569145 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.569139    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 19:24:47.569331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.569203    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9nf99\""
Jul 27 19:24:47.569331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.569229    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 19:24:47.569331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.569233    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 19:24:47.569460 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.569363    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9v6qr\""
Jul 27 19:24:47.578855 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.578799    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:47.645005 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.644970    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6852432-508f-4243-84ef-f9aebd2e2ae3-config-volume\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.645139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645023    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c9cba3-d810-4681-b777-a3bb5abad710-cert\") pod \"ingress-canary-xrhc7\" (UID: \"39c9cba3-d810-4681-b777-a3bb5abad710\") " pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.645139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645044    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkzc\" (UniqueName: \"kubernetes.io/projected/a6852432-508f-4243-84ef-f9aebd2e2ae3-kube-api-access-gkkzc\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.645139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645072    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6852432-508f-4243-84ef-f9aebd2e2ae3-tmp-dir\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.645139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645086    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wft2b\" (UniqueName: \"kubernetes.io/projected/39c9cba3-d810-4681-b777-a3bb5abad710-kube-api-access-wft2b\") pod \"ingress-canary-xrhc7\" (UID: \"39c9cba3-d810-4681-b777-a3bb5abad710\") " pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.645139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645121    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.645389 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:47.645215    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:24:47.645389 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:47.645293    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:48.145274381 +0000 UTC m=+37.141122339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:24:47.645513 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645444    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6852432-508f-4243-84ef-f9aebd2e2ae3-tmp-dir\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.645612 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.645594    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a6852432-508f-4243-84ef-f9aebd2e2ae3-config-volume\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.649605 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.649580    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c9cba3-d810-4681-b777-a3bb5abad710-cert\") pod \"ingress-canary-xrhc7\" (UID: \"39c9cba3-d810-4681-b777-a3bb5abad710\") " pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.655999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.655969    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkzc\" (UniqueName: \"kubernetes.io/projected/a6852432-508f-4243-84ef-f9aebd2e2ae3-kube-api-access-gkkzc\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:47.656381 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.656363    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wft2b\" (UniqueName: \"kubernetes.io/projected/39c9cba3-d810-4681-b777-a3bb5abad710-kube-api-access-wft2b\") pod \"ingress-canary-xrhc7\" (UID: \"39c9cba3-d810-4681-b777-a3bb5abad710\") " pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.675825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.675795    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:47.700712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.700683    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrhc7"
Jul 27 19:24:47.838445 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.838357    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerStarted","Data":"ade572f496cfb5a02526499a17b863daa2b42557ffc7c5bad222869dd0563417"}
Jul 27 19:24:47.849490 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:47.849460    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrhc7"]
Jul 27 19:24:47.879293 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:24:47.879266    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c9cba3_d810_4681_b777_a3bb5abad710.slice/crio-4a5384657e92813eab5f717ce2f613d33da61f18282504f11d08a506d677d85d WatchSource:0}: Error finding container 4a5384657e92813eab5f717ce2f613d33da61f18282504f11d08a506d677d85d: Status 404 returned error can't find the container with id 4a5384657e92813eab5f717ce2f613d33da61f18282504f11d08a506d677d85d
Jul 27 19:24:48.147890 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:48.147799    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:48.148019 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:48.147970    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:24:48.148053 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:48.148034    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:49.14801815 +0000 UTC m=+38.143866106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:24:48.843648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:48.843433    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ae699bd-8bb1-49f1-a257-bb1f406c1d22" containerID="ade572f496cfb5a02526499a17b863daa2b42557ffc7c5bad222869dd0563417" exitCode=0
Jul 27 19:24:48.844354 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:48.843505    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerDied","Data":"ade572f496cfb5a02526499a17b863daa2b42557ffc7c5bad222869dd0563417"}
Jul 27 19:24:48.844939 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:48.844911    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrhc7" event={"ID":"39c9cba3-d810-4681-b777-a3bb5abad710","Type":"ContainerStarted","Data":"4a5384657e92813eab5f717ce2f613d33da61f18282504f11d08a506d677d85d"}
Jul 27 19:24:49.155433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:49.155388    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:49.155634 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:49.155566    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:24:49.155680 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:49.155638    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:51.155618836 +0000 UTC m=+40.151466794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:24:49.850804 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:49.850756    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ae699bd-8bb1-49f1-a257-bb1f406c1d22" containerID="dafb7b6a2675ce59c0b263b14567c7b28e1c27081a9c4205cbc438979d291f19" exitCode=0
Jul 27 19:24:49.851173 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:49.850808    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerDied","Data":"dafb7b6a2675ce59c0b263b14567c7b28e1c27081a9c4205cbc438979d291f19"}
Jul 27 19:24:50.857950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:50.857906    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t925l" event={"ID":"2ae699bd-8bb1-49f1-a257-bb1f406c1d22","Type":"ContainerStarted","Data":"b6f3d64fdb3ec5c62e04f63b96e0fe8371dc14d8f2a29543a279bf70564d8e86"}
Jul 27 19:24:50.859724 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:50.859694    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrhc7" event={"ID":"39c9cba3-d810-4681-b777-a3bb5abad710","Type":"ContainerStarted","Data":"c0ef9f9d5f5ac0d955cb8f693dc4f0348426eb4b25c75ff2810a630f524da34f"}
Jul 27 19:24:50.886022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:50.885970    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t925l" podStartSLOduration=9.556470505 podStartE2EDuration="38.885953561s" totalImagesPullingTime="29.329483056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:50.884172572 +0000 UTC m=+39.880020551" watchObservedRunningTime="2026-07-27 19:24:50.885953561 +0000 UTC m=+39.881801539"
Jul 27 19:24:50.887248 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:50.887224    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:50.902187 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:50.902146    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xrhc7" podStartSLOduration=1.581916133 podStartE2EDuration="3.902133001s" totalImagesPullingTime="2.320216868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:50.901702391 +0000 UTC m=+39.897550369" watchObservedRunningTime="2026-07-27 19:24:50.902133001 +0000 UTC m=+39.897980978"
Jul 27 19:24:50.987847 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:50.987811    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:51.088331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:51.088303    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:51.170719 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:51.170633    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:51.170831 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:51.170782    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:24:51.170881 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:51.170849    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:24:55.170834538 +0000 UTC m=+44.166682495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:24:51.189618 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:51.189585    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:51.260295 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:51.260264    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:24:55.199124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:55.199075    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:24:55.199635 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:55.199244    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:24:55.199635 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:24:55.199311    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:03.19929501 +0000 UTC m=+52.195142966 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:24:59.832822 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:24:59.832781    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:01.581132 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:01.581098    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:03.252250 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:03.252202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:25:03.252660 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:03.252359    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:25:03.252660 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:03.252423    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:19.252407453 +0000 UTC m=+68.248255413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:25:12.831389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:12.831359    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdqb5"
Jul 27 19:25:12.978668 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:12.978600    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:13.144022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:13.143936    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:17.149308 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.149267    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:25:17.149741 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.149318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:25:17.152848 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.152826    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 19:25:17.152941 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.152887    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 19:25:17.160249 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:17.160228    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 19:25:17.160303 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:17.160288    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs podName:0252630f-a0ee-4786-95ab-a6cf9dec8dec nodeName:}" failed. No retries permitted until 2026-07-27 19:26:21.160274391 +0000 UTC m=+130.156122347 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs") pod "network-metrics-daemon-s8jzv" (UID: "0252630f-a0ee-4786-95ab-a6cf9dec8dec") : secret "metrics-daemon-secret" not found
Jul 27 19:25:17.163730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.163711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/047be226-fd26-402a-99b4-ebc0dadca574-original-pull-secret\") pod \"global-pull-secret-syncer-x49q7\" (UID: \"047be226-fd26-402a-99b4-ebc0dadca574\") " pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:25:17.250149 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.250106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:25:17.253603 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.253587    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 19:25:17.264495 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.264475    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 19:25:17.274411 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.274387    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htztt\" (UniqueName: \"kubernetes.io/projected/4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf-kube-api-access-htztt\") pod \"network-check-target-p5xw5\" (UID: \"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf\") " pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:25:17.299028 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.292697    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x49q7"
Jul 27 19:25:17.299028 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.293166    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9nf99\""
Jul 27 19:25:17.299028 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.293374    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:25:17.437131 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.437058    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x49q7"]
Jul 27 19:25:17.440638 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:25:17.440599    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047be226_fd26_402a_99b4_ebc0dadca574.slice/crio-c7a2129ad6bebed6e0e35a660222537e16dd244f5d06bdb57a24e98930561ccc WatchSource:0}: Error finding container c7a2129ad6bebed6e0e35a660222537e16dd244f5d06bdb57a24e98930561ccc: Status 404 returned error can't find the container with id c7a2129ad6bebed6e0e35a660222537e16dd244f5d06bdb57a24e98930561ccc
Jul 27 19:25:17.460942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.460912    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p5xw5"]
Jul 27 19:25:17.464369 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:25:17.464344    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afff9f6_5e5e_4b1b_a8ab_892cd8de94cf.slice/crio-533f2184db107e8bf4b532f1847f875648cd47f2b1ffd142960d542250a27559 WatchSource:0}: Error finding container 533f2184db107e8bf4b532f1847f875648cd47f2b1ffd142960d542250a27559: Status 404 returned error can't find the container with id 533f2184db107e8bf4b532f1847f875648cd47f2b1ffd142960d542250a27559
Jul 27 19:25:17.926165 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.926126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p5xw5" event={"ID":"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf","Type":"ContainerStarted","Data":"533f2184db107e8bf4b532f1847f875648cd47f2b1ffd142960d542250a27559"}
Jul 27 19:25:17.927358 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:17.927335    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x49q7" event={"ID":"047be226-fd26-402a-99b4-ebc0dadca574","Type":"ContainerStarted","Data":"c7a2129ad6bebed6e0e35a660222537e16dd244f5d06bdb57a24e98930561ccc"}
Jul 27 19:25:19.263942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:19.263811    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:25:19.264376 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:19.263943    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 19:25:19.264376 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:19.264031    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls podName:a6852432-508f-4243-84ef-f9aebd2e2ae3 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:51.264014415 +0000 UTC m=+100.259862371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls") pod "dns-default-kkx66" (UID: "a6852432-508f-4243-84ef-f9aebd2e2ae3") : secret "dns-default-metrics-tls" not found
Jul 27 19:25:21.217599 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:21.217564    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:22.942714 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:22.942668    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p5xw5" event={"ID":"4afff9f6-5e5e-4b1b-a8ab-892cd8de94cf","Type":"ContainerStarted","Data":"4bd312657a3fafca84a52bd0fbea58099bd03566f604c5b2a129b9e52f3357fe"}
Jul 27 19:25:22.943183 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:22.942805    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:25:22.943981 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:22.943955    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x49q7" event={"ID":"047be226-fd26-402a-99b4-ebc0dadca574","Type":"ContainerStarted","Data":"0eeb0f4372275907060c06d7fd844acb30079d349561fe27a4fc54696f5aa017"}
Jul 27 19:25:22.964399 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:22.964353    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p5xw5" podStartSLOduration=66.448249225 podStartE2EDuration="1m10.964341102s" totalImagesPullingTime="4.516091877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:22.963935081 +0000 UTC m=+71.959783059" watchObservedRunningTime="2026-07-27 19:25:22.964341102 +0000 UTC m=+71.960189079"
Jul 27 19:25:22.987985 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:22.987929    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x49q7" podStartSLOduration=66.444231239 podStartE2EDuration="1m10.987912978s" totalImagesPullingTime="4.543681739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:22.987815378 +0000 UTC m=+71.983663345" watchObservedRunningTime="2026-07-27 19:25:22.987912978 +0000 UTC m=+71.983760961"
Jul 27 19:25:36.319561 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.319430    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"]
Jul 27 19:25:36.321873 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.321856    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.324981 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.324952    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 19:25:36.325098 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.324952    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 19:25:36.325098 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.325004    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 27 19:25:36.326140 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.326121    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x8rwm\""
Jul 27 19:25:36.327975 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.327912    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 27 19:25:36.333643 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.333624    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"]
Jul 27 19:25:36.375051 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.375023    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn6f\" (UniqueName: \"kubernetes.io/projected/8fd5bbbe-6712-4160-a12a-b318f7e0387c-kube-api-access-4hn6f\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.375170 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.375060    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8fd5bbbe-6712-4160-a12a-b318f7e0387c-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.375170 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.375120    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.427418 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.427389    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"]
Jul 27 19:25:36.429286 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.429251    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"
Jul 27 19:25:36.432431 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.432411    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zlr8k\""
Jul 27 19:25:36.432632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.432614    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 19:25:36.432707 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.432638    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:25:36.436769 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.436747    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"]
Jul 27 19:25:36.438631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.438616    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:36.441103 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.441088    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 27 19:25:36.441295 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.441283    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qjn89\""
Jul 27 19:25:36.441448 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.441435    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:25:36.441781 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.441764    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 27 19:25:36.443086 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.443047    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"]
Jul 27 19:25:36.453689 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.453666    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"]
Jul 27 19:25:36.475464 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.475441    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxk6\" (UniqueName: \"kubernetes.io/projected/b188229e-a7bc-4075-b4f9-dff7763eff82-kube-api-access-wdxk6\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:36.475621 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.475477    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn6f\" (UniqueName: \"kubernetes.io/projected/8fd5bbbe-6712-4160-a12a-b318f7e0387c-kube-api-access-4hn6f\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.475621 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.475500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drmwk\" (UniqueName: \"kubernetes.io/projected/c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d-kube-api-access-drmwk\") pod \"volume-data-source-validator-549c6d54cf-sgscc\" (UID: \"c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"
Jul 27 19:25:36.475621 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.475549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:36.475786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.475627    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8fd5bbbe-6712-4160-a12a-b318f7e0387c-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.475786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.475688    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.475901 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.475794    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:36.475901 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.475887    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls podName:8fd5bbbe-6712-4160-a12a-b318f7e0387c nodeName:}" failed. No retries permitted until 2026-07-27 19:25:36.975872218 +0000 UTC m=+85.971720173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lbh6d" (UID: "8fd5bbbe-6712-4160-a12a-b318f7e0387c") : secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:36.476342 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.476326    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8fd5bbbe-6712-4160-a12a-b318f7e0387c-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.486140 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.486120    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn6f\" (UniqueName: \"kubernetes.io/projected/8fd5bbbe-6712-4160-a12a-b318f7e0387c-kube-api-access-4hn6f\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.538097 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.538066    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-459pm"]
Jul 27 19:25:36.540136 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.540120    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-96d67c84-tfbsz"]
Jul 27 19:25:36.540268 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.540254    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.541776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.541759    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.543459 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.543442    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 19:25:36.543576 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.543500    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 19:25:36.543576 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.543564    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-58dt6\""
Jul 27 19:25:36.543783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.543768    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:25:36.543981 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.543963    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 19:25:36.544290 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.544273    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 19:25:36.544612 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.544598    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 19:25:36.544687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.544643    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mlfrd\""
Jul 27 19:25:36.544842 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.544830    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 19:25:36.555174 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.555152    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-459pm"]
Jul 27 19:25:36.555834 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.555805    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 19:25:36.555961 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.555835    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 19:25:36.559639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.559618    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-96d67c84-tfbsz"]
Jul 27 19:25:36.576943 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.576880    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-trusted-ca\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.576943 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.576932    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47pj\" (UniqueName: \"kubernetes.io/projected/0bcb2391-77ca-4b27-854a-e63d209d7def-kube-api-access-t47pj\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.577091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.576955    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-image-registry-private-configuration\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.576984    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-certificates\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-bound-sa-token\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577019    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxk6\" (UniqueName: \"kubernetes.io/projected/b188229e-a7bc-4075-b4f9-dff7763eff82-kube-api-access-wdxk6\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:36.577091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577052    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drmwk\" (UniqueName: \"kubernetes.io/projected/c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d-kube-api-access-drmwk\") pod \"volume-data-source-validator-549c6d54cf-sgscc\" (UID: \"c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"
Jul 27 19:25:36.577091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577075    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ae04be-3837-4b32-b0ad-fa36f32fb045-ca-trust-extracted\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577095    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577111    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bcb2391-77ca-4b27-854a-e63d209d7def-serving-cert\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577152    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bcb2391-77ca-4b27-854a-e63d209d7def-trusted-ca\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.577245    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577262    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577288    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-installation-pull-secrets\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577315    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb2391-77ca-4b27-854a-e63d209d7def-config\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.577339    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7g6r\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-kube-api-access-d7g6r\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.577361 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.577357    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls podName:b188229e-a7bc-4075-b4f9-dff7763eff82 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:37.077336178 +0000 UTC m=+86.073184140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls") pod "cluster-samples-operator-6748489656-bkksg" (UID: "b188229e-a7bc-4075-b4f9-dff7763eff82") : secret "samples-operator-tls" not found
Jul 27 19:25:36.599451 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.599428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxk6\" (UniqueName: \"kubernetes.io/projected/b188229e-a7bc-4075-b4f9-dff7763eff82-kube-api-access-wdxk6\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:36.604346 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.604322    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drmwk\" (UniqueName: \"kubernetes.io/projected/c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d-kube-api-access-drmwk\") pod \"volume-data-source-validator-549c6d54cf-sgscc\" (UID: \"c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"
Jul 27 19:25:36.678206 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678180    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ae04be-3837-4b32-b0ad-fa36f32fb045-ca-trust-extracted\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678222    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bcb2391-77ca-4b27-854a-e63d209d7def-serving-cert\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678239    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bcb2391-77ca-4b27-854a-e63d209d7def-trusted-ca\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678253    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678275    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-installation-pull-secrets\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678294    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb2391-77ca-4b27-854a-e63d209d7def-config\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678310    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7g6r\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-kube-api-access-d7g6r\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678361 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678361    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-trusted-ca\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.678370    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.678392    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-96d67c84-tfbsz: secret "image-registry-tls" not found
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678424    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47pj\" (UniqueName: \"kubernetes.io/projected/0bcb2391-77ca-4b27-854a-e63d209d7def-kube-api-access-t47pj\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.678460    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls podName:39ae04be-3837-4b32-b0ad-fa36f32fb045 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:37.17844128 +0000 UTC m=+86.174289257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls") pod "image-registry-96d67c84-tfbsz" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045") : secret "image-registry-tls" not found
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-image-registry-private-configuration\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678556    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-certificates\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678586    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-bound-sa-token\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.678742 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.678616    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ae04be-3837-4b32-b0ad-fa36f32fb045-ca-trust-extracted\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.679294 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.679134    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb2391-77ca-4b27-854a-e63d209d7def-config\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.679294 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.679186    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bcb2391-77ca-4b27-854a-e63d209d7def-trusted-ca\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.679400 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.679339    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-certificates\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.679646 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.679617    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-trusted-ca\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.680830 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.680811    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bcb2391-77ca-4b27-854a-e63d209d7def-serving-cert\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.680918 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.680867    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-installation-pull-secrets\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.681602 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.681586    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-image-registry-private-configuration\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.689115 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.689092    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47pj\" (UniqueName: \"kubernetes.io/projected/0bcb2391-77ca-4b27-854a-e63d209d7def-kube-api-access-t47pj\") pod \"console-operator-584dcb6b54-459pm\" (UID: \"0bcb2391-77ca-4b27-854a-e63d209d7def\") " pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.689617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.689603    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-bound-sa-token\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.690161 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.690147    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7g6r\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-kube-api-access-d7g6r\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:36.738061 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.738032    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"
Jul 27 19:25:36.760740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.760713    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:36.841353 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.841248    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:36.854205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.854165    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:36.867786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.867763    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc"]
Jul 27 19:25:36.871390 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:25:36.871365    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ae37b7_a0e5_4d21_af44_03c07bdc6e5d.slice/crio-b3d96e9f6e634eebb2b930161a9d205d5fa3cb45743e989f9ac6a197d8d0c0c7 WatchSource:0}: Error finding container b3d96e9f6e634eebb2b930161a9d205d5fa3cb45743e989f9ac6a197d8d0c0c7: Status 404 returned error can't find the container with id b3d96e9f6e634eebb2b930161a9d205d5fa3cb45743e989f9ac6a197d8d0c0c7
Jul 27 19:25:36.981211 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.981177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:36.981381 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.981178    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc" event={"ID":"c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d","Type":"ContainerStarted","Data":"b3d96e9f6e634eebb2b930161a9d205d5fa3cb45743e989f9ac6a197d8d0c0c7"}
Jul 27 19:25:36.981381 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.981278    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:36.981381 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:36.981380    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls podName:8fd5bbbe-6712-4160-a12a-b318f7e0387c nodeName:}" failed. No retries permitted until 2026-07-27 19:25:37.981358082 +0000 UTC m=+86.977206057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lbh6d" (UID: "8fd5bbbe-6712-4160-a12a-b318f7e0387c") : secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:36.982767 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:36.982744    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-459pm"]
Jul 27 19:25:36.985933 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:25:36.985914    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcb2391_77ca_4b27_854a_e63d209d7def.slice/crio-defea132e8864292215706d58e6efed08ca602a00b2b90f98826411abd2d7491 WatchSource:0}: Error finding container defea132e8864292215706d58e6efed08ca602a00b2b90f98826411abd2d7491: Status 404 returned error can't find the container with id defea132e8864292215706d58e6efed08ca602a00b2b90f98826411abd2d7491
Jul 27 19:25:37.081648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:37.081612    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:37.081819 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.081756    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 19:25:37.081883 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.081820    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls podName:b188229e-a7bc-4075-b4f9-dff7763eff82 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:38.081804555 +0000 UTC m=+87.077652510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls") pod "cluster-samples-operator-6748489656-bkksg" (UID: "b188229e-a7bc-4075-b4f9-dff7763eff82") : secret "samples-operator-tls" not found
Jul 27 19:25:37.182486 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:37.182410    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:37.182634 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.182618    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:25:37.182675 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.182637    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-96d67c84-tfbsz: secret "image-registry-tls" not found
Jul 27 19:25:37.182707 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.182687    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls podName:39ae04be-3837-4b32-b0ad-fa36f32fb045 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:38.182674083 +0000 UTC m=+87.178522039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls") pod "image-registry-96d67c84-tfbsz" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045") : secret "image-registry-tls" not found
Jul 27 19:25:37.985701 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:37.985665    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" event={"ID":"0bcb2391-77ca-4b27-854a-e63d209d7def","Type":"ContainerStarted","Data":"defea132e8864292215706d58e6efed08ca602a00b2b90f98826411abd2d7491"}
Jul 27 19:25:37.988174 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:37.988138    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:37.988325 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.988301    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:37.988403 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:37.988396    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls podName:8fd5bbbe-6712-4160-a12a-b318f7e0387c nodeName:}" failed. No retries permitted until 2026-07-27 19:25:39.988375201 +0000 UTC m=+88.984223162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lbh6d" (UID: "8fd5bbbe-6712-4160-a12a-b318f7e0387c") : secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:38.089507 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:38.089480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:38.089697 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:38.089649    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 19:25:38.089773 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:38.089725    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls podName:b188229e-a7bc-4075-b4f9-dff7763eff82 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:40.089704378 +0000 UTC m=+89.085552340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls") pod "cluster-samples-operator-6748489656-bkksg" (UID: "b188229e-a7bc-4075-b4f9-dff7763eff82") : secret "samples-operator-tls" not found
Jul 27 19:25:38.190262 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:38.190229    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:38.190448 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:38.190414    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:25:38.190448 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:38.190437    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-96d67c84-tfbsz: secret "image-registry-tls" not found
Jul 27 19:25:38.190589 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:38.190508    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls podName:39ae04be-3837-4b32-b0ad-fa36f32fb045 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:40.190487927 +0000 UTC m=+89.186335905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls") pod "image-registry-96d67c84-tfbsz" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045") : secret "image-registry-tls" not found
Jul 27 19:25:38.990008 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:38.989973    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc" event={"ID":"c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d","Type":"ContainerStarted","Data":"8bdff4f18c971a7d9db1e966019294c40c84177ceb6bc43f3d33c5d2a5cc2b13"}
Jul 27 19:25:39.012647 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.012600    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sgscc" podStartSLOduration=1.826327439 podStartE2EDuration="3.012582843s" totalImagesPullingTime="1.186255404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:39.012001907 +0000 UTC m=+88.007849886" watchObservedRunningTime="2026-07-27 19:25:39.012582843 +0000 UTC m=+88.008430848"
Jul 27 19:25:39.137782 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.137745    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:39.239039 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.238999    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:39.993975 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.993942    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/0.log"
Jul 27 19:25:39.994446 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.993987    2568 generic.go:356] "Generic (PLEG): container finished" podID="0bcb2391-77ca-4b27-854a-e63d209d7def" containerID="ea94a849d57704ba1f9ffec84a2bdfd4632b03051c2403b19f30815cec2a20ae" exitCode=255
Jul 27 19:25:39.994446 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.994080    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" event={"ID":"0bcb2391-77ca-4b27-854a-e63d209d7def","Type":"ContainerDied","Data":"ea94a849d57704ba1f9ffec84a2bdfd4632b03051c2403b19f30815cec2a20ae"}
Jul 27 19:25:39.994446 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:39.994247    2568 scope.go:117] "RemoveContainer" containerID="ea94a849d57704ba1f9ffec84a2bdfd4632b03051c2403b19f30815cec2a20ae"
Jul 27 19:25:40.004671 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.004651    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:40.004777 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.004765    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:40.004833 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.004823    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls podName:8fd5bbbe-6712-4160-a12a-b318f7e0387c nodeName:}" failed. No retries permitted until 2026-07-27 19:25:44.004809382 +0000 UTC m=+93.000657341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lbh6d" (UID: "8fd5bbbe-6712-4160-a12a-b318f7e0387c") : secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:40.105201 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.105176    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:40.105333 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.105312    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 19:25:40.105412 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.105401    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls podName:b188229e-a7bc-4075-b4f9-dff7763eff82 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:44.105379904 +0000 UTC m=+93.101227889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls") pod "cluster-samples-operator-6748489656-bkksg" (UID: "b188229e-a7bc-4075-b4f9-dff7763eff82") : secret "samples-operator-tls" not found
Jul 27 19:25:40.206412 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.206373    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:40.206593 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.206544    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:25:40.206593 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.206569    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-96d67c84-tfbsz: secret "image-registry-tls" not found
Jul 27 19:25:40.206661 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.206623    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls podName:39ae04be-3837-4b32-b0ad-fa36f32fb045 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:44.206606856 +0000 UTC m=+93.202454812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls") pod "image-registry-96d67c84-tfbsz" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045") : secret "image-registry-tls" not found
Jul 27 19:25:40.385372 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.385318    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:40.447018 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.446989    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mbvtm"]
Jul 27 19:25:40.449026 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.449011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.451792 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.451770    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 19:25:40.451916 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.451792    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 19:25:40.451916 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.451792    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 19:25:40.451916 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.451778    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 19:25:40.452141 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.452129    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kpwnj\""
Jul 27 19:25:40.462822 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.462800    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mbvtm"]
Jul 27 19:25:40.473851 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.473833    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:40.509352 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.509331    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.509469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.509357    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75395a11-454f-4e19-bd8d-842ccda55d1d-data-volume\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.509469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.509397    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75395a11-454f-4e19-bd8d-842ccda55d1d-crio-socket\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.509469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.509449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.509626 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.509505    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrc2x\" (UniqueName: \"kubernetes.io/projected/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-api-access-wrc2x\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.555338 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.555314    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:40.610483 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610455    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75395a11-454f-4e19-bd8d-842ccda55d1d-crio-socket\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.610631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610493    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.610631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610546    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrc2x\" (UniqueName: \"kubernetes.io/projected/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-api-access-wrc2x\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.610631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610589    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.610631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610614    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75395a11-454f-4e19-bd8d-842ccda55d1d-data-volume\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.610779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610592    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75395a11-454f-4e19-bd8d-842ccda55d1d-crio-socket\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.610779 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.610658    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:25:40.610779 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.610697    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:40.610779 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.610728    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:41.11070826 +0000 UTC m=+90.106556234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : configmap "kube-rbac-proxy" not found
Jul 27 19:25:40.610779 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.610766    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:41.110754152 +0000 UTC m=+90.106602107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:40.610961 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.610942    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75395a11-454f-4e19-bd8d-842ccda55d1d-data-volume\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.623356 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.623333    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrc2x\" (UniqueName: \"kubernetes.io/projected/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-api-access-wrc2x\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:40.857969 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.857938    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:40.998195 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.998169    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/1.log"
Jul 27 19:25:40.998639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.998609    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/0.log"
Jul 27 19:25:40.998704 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.998651    2568 generic.go:356] "Generic (PLEG): container finished" podID="0bcb2391-77ca-4b27-854a-e63d209d7def" containerID="0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086" exitCode=255
Jul 27 19:25:40.998755 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.998719    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" event={"ID":"0bcb2391-77ca-4b27-854a-e63d209d7def","Type":"ContainerDied","Data":"0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086"}
Jul 27 19:25:40.998811 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.998778    2568 scope.go:117] "RemoveContainer" containerID="ea94a849d57704ba1f9ffec84a2bdfd4632b03051c2403b19f30815cec2a20ae"
Jul 27 19:25:40.999039 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:40.999023    2568 scope.go:117] "RemoveContainer" containerID="0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086"
Jul 27 19:25:40.999201 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:40.999183    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-459pm_openshift-console-operator(0bcb2391-77ca-4b27-854a-e63d209d7def)\"" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podUID="0bcb2391-77ca-4b27-854a-e63d209d7def"
Jul 27 19:25:41.115648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:41.115542    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:41.115648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:41.115638    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:41.115901 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:41.115669    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:25:41.115901 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:41.115739    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:42.115719083 +0000 UTC m=+91.111567041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : configmap "kube-rbac-proxy" not found
Jul 27 19:25:41.115901 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:41.115806    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:41.115901 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:41.115866    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:42.115850671 +0000 UTC m=+91.111698627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:41.225651 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:41.225617    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:41.461564 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:41.461473    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:41.532409 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:41.532383    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:42.002759 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:42.002733    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/1.log"
Jul 27 19:25:42.003126 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:42.003038    2568 scope.go:117] "RemoveContainer" containerID="0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086"
Jul 27 19:25:42.003197 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:42.003177    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-459pm_openshift-console-operator(0bcb2391-77ca-4b27-854a-e63d209d7def)\"" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podUID="0bcb2391-77ca-4b27-854a-e63d209d7def"
Jul 27 19:25:42.123965 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:42.123927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:42.124143 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:42.124060    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:25:42.124143 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:42.124133    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:44.124112097 +0000 UTC m=+93.119960068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : configmap "kube-rbac-proxy" not found
Jul 27 19:25:42.124241 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:42.124217    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:42.124380 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:42.124363    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:42.124434 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:42.124418    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:44.124400782 +0000 UTC m=+93.120248739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:42.617203 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:42.617174    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-slvtt_40900963-6214-4d92-8e58-7f7ab9fff63a/dns-node-resolver/0.log"
Jul 27 19:25:43.135606 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:43.135570    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:43.216381 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:43.216346    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:43.419967 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:43.419893    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcp6x_61ab69d2-8341-4561-8ed0-9873bff1eec1/node-ca/0.log"
Jul 27 19:25:44.039671 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.039624    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:44.039869 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.039744    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:44.039869 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.039802    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls podName:8fd5bbbe-6712-4160-a12a-b318f7e0387c nodeName:}" failed. No retries permitted until 2026-07-27 19:25:52.039786963 +0000 UTC m=+101.035634919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lbh6d" (UID: "8fd5bbbe-6712-4160-a12a-b318f7e0387c") : secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:44.140893 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.140855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.140912    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.140939    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.141010    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.141008    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.141053    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.141062    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:48.141049013 +0000 UTC m=+97.136896968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : configmap "kube-rbac-proxy" not found
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.141074    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:48.141068427 +0000 UTC m=+97.136916383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:44.141284 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.141090    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls podName:b188229e-a7bc-4075-b4f9-dff7763eff82 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:52.141078017 +0000 UTC m=+101.136925973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls") pod "cluster-samples-operator-6748489656-bkksg" (UID: "b188229e-a7bc-4075-b4f9-dff7763eff82") : secret "samples-operator-tls" not found
Jul 27 19:25:44.242246 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.242204    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:44.242399 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.242351    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 19:25:44.242399 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.242371    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-96d67c84-tfbsz: secret "image-registry-tls" not found
Jul 27 19:25:44.242470 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:44.242422    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls podName:39ae04be-3837-4b32-b0ad-fa36f32fb045 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:52.242407576 +0000 UTC m=+101.238255532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls") pod "image-registry-96d67c84-tfbsz" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045") : secret "image-registry-tls" not found
Jul 27 19:25:44.384713 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.384635    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:44.417369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:44.417345    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrhc7_39c9cba3-d810-4681-b777-a3bb5abad710/serve-healthcheck-canary/0.log"
Jul 27 19:25:46.814714 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:46.814675    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:46.855324 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:46.855299    2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:46.855424 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:46.855347    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:25:46.855670 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:46.855659    2568 scope.go:117] "RemoveContainer" containerID="0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086"
Jul 27 19:25:46.855809 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:46.855792    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-459pm_openshift-console-operator(0bcb2391-77ca-4b27-854a-e63d209d7def)\"" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podUID="0bcb2391-77ca-4b27-854a-e63d209d7def"
Jul 27 19:25:46.933924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:46.933888    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:47.038503 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:47.038472    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:47.175600 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:47.175507    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:47.277359 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:47.277332    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:47.403228 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:47.403198    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:47.533210 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:47.533183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:47.758712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:47.758680    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:48.172276 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:48.172240    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:48.172712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:48.172321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:48.172712 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:48.172360    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 19:25:48.172712 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:48.172424    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:56.172409693 +0000 UTC m=+105.168257648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : configmap "kube-rbac-proxy" not found
Jul 27 19:25:48.172712 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:48.172437    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:48.172712 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:48.172490    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:25:56.172476482 +0000 UTC m=+105.168324441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:48.315463 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:48.315431    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:48.930330 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:48.930296    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:49.121031 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:49.120994    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:49.320838 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:49.320807    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:49.712149 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:49.712065    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:50.114501 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:50.114471    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:50.561628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:50.561596    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:50.643951 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:50.643924    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:50.724392 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:50.724363    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:51.140840 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.140808    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:51.242032 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.242006    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:51.299727 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.299700    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:25:51.302119 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.302101    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a6852432-508f-4243-84ef-f9aebd2e2ae3-metrics-tls\") pod \"dns-default-kkx66\" (UID: \"a6852432-508f-4243-84ef-f9aebd2e2ae3\") " pod="openshift-dns/dns-default-kkx66"
Jul 27 19:25:51.311133 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.311113    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksszw\""
Jul 27 19:25:51.318740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.318725    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kkx66"
Jul 27 19:25:51.450214 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.450185    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kkx66"]
Jul 27 19:25:51.456410 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:25:51.456385    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6852432_508f_4243_84ef_f9aebd2e2ae3.slice/crio-669e9104097ea2b85a8fd8ddb1ad886db7e6b6c9156ba79bab570113e23b17ea WatchSource:0}: Error finding container 669e9104097ea2b85a8fd8ddb1ad886db7e6b6c9156ba79bab570113e23b17ea: Status 404 returned error can't find the container with id 669e9104097ea2b85a8fd8ddb1ad886db7e6b6c9156ba79bab570113e23b17ea
Jul 27 19:25:51.516047 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.516008    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:51.919689 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:51.919655    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:52.031252 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.031219    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kkx66" event={"ID":"a6852432-508f-4243-84ef-f9aebd2e2ae3","Type":"ContainerStarted","Data":"669e9104097ea2b85a8fd8ddb1ad886db7e6b6c9156ba79bab570113e23b17ea"}
Jul 27 19:25:52.106703 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.106668    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:25:52.106892 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:52.106858    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:52.106945 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:52.106939    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls podName:8fd5bbbe-6712-4160-a12a-b318f7e0387c nodeName:}" failed. No retries permitted until 2026-07-27 19:26:08.106918803 +0000 UTC m=+117.102766762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-lbh6d" (UID: "8fd5bbbe-6712-4160-a12a-b318f7e0387c") : secret "cluster-monitoring-operator-tls" not found
Jul 27 19:25:52.119365 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.119334    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:52.207419 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.207347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:52.210011 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.209977    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b188229e-a7bc-4075-b4f9-dff7763eff82-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-bkksg\" (UID: \"b188229e-a7bc-4075-b4f9-dff7763eff82\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:52.308601 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.308564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:52.311446 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.311421    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"image-registry-96d67c84-tfbsz\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") " pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:52.349004 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.348980    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"
Jul 27 19:25:52.460267 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.460187    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:52.524167 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.524134    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg"]
Jul 27 19:25:52.604473 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.604431    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-96d67c84-tfbsz"]
Jul 27 19:25:52.606912 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:25:52.606877    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ae04be_3837_4b32_b0ad_fa36f32fb045.slice/crio-0654ae2beae07ca8cc135990c2f3c65bd0d65e0d91d94c41b1e242f123f53ffa WatchSource:0}: Error finding container 0654ae2beae07ca8cc135990c2f3c65bd0d65e0d91d94c41b1e242f123f53ffa: Status 404 returned error can't find the container with id 0654ae2beae07ca8cc135990c2f3c65bd0d65e0d91d94c41b1e242f123f53ffa
Jul 27 19:25:52.724424 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.724349    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:52.824743 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:52.824697    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:53.036308 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.036269    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg" event={"ID":"b188229e-a7bc-4075-b4f9-dff7763eff82","Type":"ContainerStarted","Data":"5cfdde24e9942cb07e526b2a3b239f64c6ad1328022a344b58bc5b503954c879"}
Jul 27 19:25:53.038078 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.038050    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-96d67c84-tfbsz" event={"ID":"39ae04be-3837-4b32-b0ad-fa36f32fb045","Type":"ContainerStarted","Data":"bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc"}
Jul 27 19:25:53.038205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.038101    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-96d67c84-tfbsz" event={"ID":"39ae04be-3837-4b32-b0ad-fa36f32fb045","Type":"ContainerStarted","Data":"0654ae2beae07ca8cc135990c2f3c65bd0d65e0d91d94c41b1e242f123f53ffa"}
Jul 27 19:25:53.038300 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.038284    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:25:53.068997 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.068939    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-96d67c84-tfbsz" podStartSLOduration=17.068923536 podStartE2EDuration="17.068923536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:53.067494522 +0000 UTC m=+102.063342511" watchObservedRunningTime="2026-07-27 19:25:53.068923536 +0000 UTC m=+102.064771513"
Jul 27 19:25:53.073126 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.073101    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:53.334084 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.334018    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:53.519398 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.519366    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:53.617836 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.617804    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:53.925599 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.925540    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:53.949340 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:53.949310    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p5xw5"
Jul 27 19:25:54.043491 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:54.043456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kkx66" event={"ID":"a6852432-508f-4243-84ef-f9aebd2e2ae3","Type":"ContainerStarted","Data":"0d0fa83635942ea059b4b96f3f402877c2ea8bd7c9719deaad49321b3da94e39"}
Jul 27 19:25:54.043918 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:54.043513    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kkx66" event={"ID":"a6852432-508f-4243-84ef-f9aebd2e2ae3","Type":"ContainerStarted","Data":"c6d1d99757f9e38518c776adb2bc45227c21973a10a6e6b1f0588303b2b144aa"}
Jul 27 19:25:54.073397 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:54.073338    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kkx66" podStartSLOduration=64.942858949 podStartE2EDuration="1m7.073316047s" totalImagesPullingTime="2.130457098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:54.071869926 +0000 UTC m=+103.067717903" watchObservedRunningTime="2026-07-27 19:25:54.073316047 +0000 UTC m=+103.069164028"
Jul 27 19:25:54.124383 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:54.124259    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:54.318659 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:54.318625    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:55.068542 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:55.068495    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kkx66"
Jul 27 19:25:55.319874 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:55.319841    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:55.523479 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:55.523448    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:55.863977 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:55.863948    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:55.963575 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:55.963544    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:56.051699 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.051668    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg" event={"ID":"b188229e-a7bc-4075-b4f9-dff7763eff82","Type":"ContainerStarted","Data":"ac56746d23335e30555c39cecb236710f51e0b13fb8087b8c20011caff83c828"}
Jul 27 19:25:56.051876 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.051732    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg" event={"ID":"b188229e-a7bc-4075-b4f9-dff7763eff82","Type":"ContainerStarted","Data":"31194e4fff8e3122dc4938952b7b058cd38d66d18d962d597cb52c44eeb8b44e"}
Jul 27 19:25:56.082973 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.082925    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-bkksg" podStartSLOduration=17.651943607 podStartE2EDuration="20.082909529s" totalImagesPullingTime="2.430965922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:56.082280859 +0000 UTC m=+105.078128837" watchObservedRunningTime="2026-07-27 19:25:56.082909529 +0000 UTC m=+105.078757507"
Jul 27 19:25:56.125566 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.125489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:56.240360 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.240327    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:56.240557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.240402    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:56.240557 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:56.240496    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:56.240692 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:25:56.240573    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls podName:75395a11-454f-4e19-bd8d-842ccda55d1d nodeName:}" failed. No retries permitted until 2026-07-27 19:26:12.240558722 +0000 UTC m=+121.236406678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mbvtm" (UID: "75395a11-454f-4e19-bd8d-842ccda55d1d") : secret "insights-runtime-extractor-tls" not found
Jul 27 19:25:56.240968 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.240947    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75395a11-454f-4e19-bd8d-842ccda55d1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:25:56.320641 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:56.320608    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:57.173064 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:57.173027    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:57.326095 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:57.326065    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:57.511585 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:57.511488    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:57.709947 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:57.709903    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:25:57.807592 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:25:57.807565    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:00.561442 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:00.561411    2568 scope.go:117] "RemoveContainer" containerID="0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086"
Jul 27 19:26:01.068224 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:01.068196    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:26:01.068600 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:01.068583    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/1.log"
Jul 27 19:26:01.068713 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:01.068623    2568 generic.go:356] "Generic (PLEG): container finished" podID="0bcb2391-77ca-4b27-854a-e63d209d7def" containerID="9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495" exitCode=255
Jul 27 19:26:01.068713 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:01.068696    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" event={"ID":"0bcb2391-77ca-4b27-854a-e63d209d7def","Type":"ContainerDied","Data":"9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495"}
Jul 27 19:26:01.068813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:01.068750    2568 scope.go:117] "RemoveContainer" containerID="0552e8d025a794e46e7b4787d388d97aa7b011ef88daf3df76cad2910a36c086"
Jul 27 19:26:01.069079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:01.069056    2568 scope.go:117] "RemoveContainer" containerID="9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495"
Jul 27 19:26:01.069265 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:01.069246    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-459pm_openshift-console-operator(0bcb2391-77ca-4b27-854a-e63d209d7def)\"" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podUID="0bcb2391-77ca-4b27-854a-e63d209d7def"
Jul 27 19:26:02.072986 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:02.072959    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:26:05.054077 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:05.054044    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kkx66"
Jul 27 19:26:05.855222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:05.855188    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:06.854972 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.854925    2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:06.855463 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.855003    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:06.855463 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.855374    2568 scope.go:117] "RemoveContainer" containerID="9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495"
Jul 27 19:26:06.855628 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:06.855603    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-459pm_openshift-console-operator(0bcb2391-77ca-4b27-854a-e63d209d7def)\"" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podUID="0bcb2391-77ca-4b27-854a-e63d209d7def"
Jul 27 19:26:06.920028 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.919996    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"]
Jul 27 19:26:06.922620 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.922600    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:06.925990 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.925966    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 19:26:06.926103 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.926043    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lbdfg\""
Jul 27 19:26:06.926103 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.926048    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 19:26:06.926919 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.926902    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 19:26:06.927575 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.927563    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 19:26:06.936803 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.936781    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"]
Jul 27 19:26:06.942415 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.942394    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-96d67c84-tfbsz"]
Jul 27 19:26:06.996351 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.996324    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69f478d788-xt6q7"]
Jul 27 19:26:06.998401 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:06.998381    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.015801 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.015782    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f478d788-xt6q7"]
Jul 27 19:26:07.026033 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.026012    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bc8006f-f2bf-4626-8d48-85001cb429d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f4cbd988d-ds9tv\" (UID: \"1bc8006f-f2bf-4626-8d48-85001cb429d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.026149 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.026052    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt8sc\" (UniqueName: \"kubernetes.io/projected/1bc8006f-f2bf-4626-8d48-85001cb429d8-kube-api-access-dt8sc\") pod \"managed-serviceaccount-addon-agent-f4cbd988d-ds9tv\" (UID: \"1bc8006f-f2bf-4626-8d48-85001cb429d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.033250 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.033231    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:07.127175 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127083    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-registry-tls\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127175 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bc8006f-f2bf-4626-8d48-85001cb429d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f4cbd988d-ds9tv\" (UID: \"1bc8006f-f2bf-4626-8d48-85001cb429d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.127175 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbdd8b60-768e-4803-92fa-c2bae4520a55-installation-pull-secrets\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127407 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127193    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbdd8b60-768e-4803-92fa-c2bae4520a55-registry-certificates\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127407 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbdd8b60-768e-4803-92fa-c2bae4520a55-image-registry-private-configuration\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127407 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lk4\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-kube-api-access-p9lk4\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127407 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127295    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt8sc\" (UniqueName: \"kubernetes.io/projected/1bc8006f-f2bf-4626-8d48-85001cb429d8-kube-api-access-dt8sc\") pod \"managed-serviceaccount-addon-agent-f4cbd988d-ds9tv\" (UID: \"1bc8006f-f2bf-4626-8d48-85001cb429d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.127407 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127362    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd8b60-768e-4803-92fa-c2bae4520a55-trusted-ca\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127407 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127401    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-bound-sa-token\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.127651 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.127439    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbdd8b60-768e-4803-92fa-c2bae4520a55-ca-trust-extracted\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.129970 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.129945    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bc8006f-f2bf-4626-8d48-85001cb429d8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f4cbd988d-ds9tv\" (UID: \"1bc8006f-f2bf-4626-8d48-85001cb429d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.131802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.131780    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:07.137459 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.137429    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt8sc\" (UniqueName: \"kubernetes.io/projected/1bc8006f-f2bf-4626-8d48-85001cb429d8-kube-api-access-dt8sc\") pod \"managed-serviceaccount-addon-agent-f4cbd988d-ds9tv\" (UID: \"1bc8006f-f2bf-4626-8d48-85001cb429d8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.228802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-registry-tls\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.228976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228813    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbdd8b60-768e-4803-92fa-c2bae4520a55-installation-pull-secrets\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.228976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228837    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbdd8b60-768e-4803-92fa-c2bae4520a55-registry-certificates\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.228976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228867    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbdd8b60-768e-4803-92fa-c2bae4520a55-image-registry-private-configuration\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.228976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228897    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lk4\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-kube-api-access-p9lk4\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.228976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228935    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd8b60-768e-4803-92fa-c2bae4520a55-trusted-ca\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.228976 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228959    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-bound-sa-token\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.229292 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.228987    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbdd8b60-768e-4803-92fa-c2bae4520a55-ca-trust-extracted\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.229382 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.229360    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbdd8b60-768e-4803-92fa-c2bae4520a55-ca-trust-extracted\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.229921 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.229890    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbdd8b60-768e-4803-92fa-c2bae4520a55-registry-certificates\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.230048 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.230029    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbdd8b60-768e-4803-92fa-c2bae4520a55-trusted-ca\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.231347 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.231329    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbdd8b60-768e-4803-92fa-c2bae4520a55-image-registry-private-configuration\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.231435 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.231411    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-registry-tls\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.231507 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.231489    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbdd8b60-768e-4803-92fa-c2bae4520a55-installation-pull-secrets\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.237005 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.236969    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:07.239107 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.239090    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"
Jul 27 19:26:07.242747 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.242726    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-bound-sa-token\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.242994 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.242979    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lk4\" (UniqueName: \"kubernetes.io/projected/dbdd8b60-768e-4803-92fa-c2bae4520a55-kube-api-access-p9lk4\") pod \"image-registry-69f478d788-xt6q7\" (UID: \"dbdd8b60-768e-4803-92fa-c2bae4520a55\") " pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.307069 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.307041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:07.370876 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.370821    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv"]
Jul 27 19:26:07.373851 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:26:07.373811    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc8006f_f2bf_4626_8d48_85001cb429d8.slice/crio-aba593fcf3d6c889edb1b5a1defc90f7ed05a0b4b0606403b6f9e98794c8438f WatchSource:0}: Error finding container aba593fcf3d6c889edb1b5a1defc90f7ed05a0b4b0606403b6f9e98794c8438f: Status 404 returned error can't find the container with id aba593fcf3d6c889edb1b5a1defc90f7ed05a0b4b0606403b6f9e98794c8438f
Jul 27 19:26:07.448093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:07.448061    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69f478d788-xt6q7"]
Jul 27 19:26:07.450912 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:26:07.450876    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdd8b60_768e_4803_92fa_c2bae4520a55.slice/crio-e0a4de7f932c1a50f0cd3d6d5360d7f7d8d0b3858d02a8106d3caa6ecbeed8d9 WatchSource:0}: Error finding container e0a4de7f932c1a50f0cd3d6d5360d7f7d8d0b3858d02a8106d3caa6ecbeed8d9: Status 404 returned error can't find the container with id e0a4de7f932c1a50f0cd3d6d5360d7f7d8d0b3858d02a8106d3caa6ecbeed8d9
Jul 27 19:26:08.094872 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.094831    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv" event={"ID":"1bc8006f-f2bf-4626-8d48-85001cb429d8","Type":"ContainerStarted","Data":"aba593fcf3d6c889edb1b5a1defc90f7ed05a0b4b0606403b6f9e98794c8438f"}
Jul 27 19:26:08.096624 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.096596    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f478d788-xt6q7" event={"ID":"dbdd8b60-768e-4803-92fa-c2bae4520a55","Type":"ContainerStarted","Data":"037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626"}
Jul 27 19:26:08.096757 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.096642    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69f478d788-xt6q7" event={"ID":"dbdd8b60-768e-4803-92fa-c2bae4520a55","Type":"ContainerStarted","Data":"e0a4de7f932c1a50f0cd3d6d5360d7f7d8d0b3858d02a8106d3caa6ecbeed8d9"}
Jul 27 19:26:08.096819 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.096804    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:08.129004 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.128935    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69f478d788-xt6q7" podStartSLOduration=2.128918502 podStartE2EDuration="2.128918502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:08.128685992 +0000 UTC m=+117.124533971" watchObservedRunningTime="2026-07-27 19:26:08.128918502 +0000 UTC m=+117.124766458"
Jul 27 19:26:08.135217 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.135184    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:26:08.138617 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.138589    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8fd5bbbe-6712-4160-a12a-b318f7e0387c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-lbh6d\" (UID: \"8fd5bbbe-6712-4160-a12a-b318f7e0387c\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:26:08.430304 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.430220    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"
Jul 27 19:26:08.618380 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:08.618351    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d"]
Jul 27 19:26:08.621405 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:26:08.621372    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd5bbbe_6712_4160_a12a_b318f7e0387c.slice/crio-9af05adaaec12580ad8bae23f2ab227c3c5049a4a55605b1095c57968a049159 WatchSource:0}: Error finding container 9af05adaaec12580ad8bae23f2ab227c3c5049a4a55605b1095c57968a049159: Status 404 returned error can't find the container with id 9af05adaaec12580ad8bae23f2ab227c3c5049a4a55605b1095c57968a049159
Jul 27 19:26:09.101855 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:09.101815    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d" event={"ID":"8fd5bbbe-6712-4160-a12a-b318f7e0387c","Type":"ContainerStarted","Data":"9af05adaaec12580ad8bae23f2ab227c3c5049a4a55605b1095c57968a049159"}
Jul 27 19:26:10.106471 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:10.106385    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv" event={"ID":"1bc8006f-f2bf-4626-8d48-85001cb429d8","Type":"ContainerStarted","Data":"d8eb4af2d7d4de8ee73289bd55a982d76541eeb02accb930cc00f3d9a5af9356"}
Jul 27 19:26:10.132073 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:10.128822    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f4cbd988d-ds9tv" podStartSLOduration=1.6931637240000001 podStartE2EDuration="4.128760473s" totalImagesPullingTime="2.435596749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:10.126893198 +0000 UTC m=+119.122741176" watchObservedRunningTime="2026-07-27 19:26:10.128760473 +0000 UTC m=+119.124608452"
Jul 27 19:26:11.111320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:11.111278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d" event={"ID":"8fd5bbbe-6712-4160-a12a-b318f7e0387c","Type":"ContainerStarted","Data":"33d42e4d69727063a6c72fc46e8454a43d2cfc2299e801db6894a33abd78658c"}
Jul 27 19:26:11.302879 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:11.302847    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:11.460699 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:11.460673    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:11.559912 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:11.559798    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:11.914321 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:11.914234    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.091714 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.091675    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.174448 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.174362    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.271969 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.271933    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:12.274242 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.274222    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75395a11-454f-4e19-bd8d-842ccda55d1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mbvtm\" (UID: \"75395a11-454f-4e19-bd8d-842ccda55d1d\") " pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:12.309627 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.309598    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.471820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.471702    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.561321 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.561293    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kpwnj\""
Jul 27 19:26:12.568175 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.568151    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:12.571067 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.571039    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.704092 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.704012    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-lbh6d" podStartSLOduration=34.310168517 podStartE2EDuration="36.703991768s" totalImagesPullingTime="2.393823251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:11.131238314 +0000 UTC m=+120.127086308" watchObservedRunningTime="2026-07-27 19:26:12.703991768 +0000 UTC m=+121.699839746"
Jul 27 19:26:12.704634 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.704613    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mbvtm"]
Jul 27 19:26:12.707289 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:26:12.707260    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75395a11_454f_4e19_bd8d_842ccda55d1d.slice/crio-2eaa67aff05c11d49d51b2f996d9c6f77876b91826aa1614112a42c7302f82bc WatchSource:0}: Error finding container 2eaa67aff05c11d49d51b2f996d9c6f77876b91826aa1614112a42c7302f82bc: Status 404 returned error can't find the container with id 2eaa67aff05c11d49d51b2f996d9c6f77876b91826aa1614112a42c7302f82bc
Jul 27 19:26:12.709170 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.709145    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:12.812306 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:12.812276    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:13.119734 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:13.119656    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbvtm" event={"ID":"75395a11-454f-4e19-bd8d-842ccda55d1d","Type":"ContainerStarted","Data":"2eaa67aff05c11d49d51b2f996d9c6f77876b91826aa1614112a42c7302f82bc"}
Jul 27 19:26:14.614038 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:14.613999    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:14.711102 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:14.711071    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:14.968167 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:14.968083    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:15.107481 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:15.107448    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:15.207335 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:15.207300    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:15.308078 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:15.308047    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:16.058885 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:16.058849    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:16.153241 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:16.153211    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:16.950037 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:16.950005    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:16.973738 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:16.973674    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:18.142622 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.142582    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbvtm" event={"ID":"75395a11-454f-4e19-bd8d-842ccda55d1d","Type":"ContainerStarted","Data":"d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b"}
Jul 27 19:26:18.162079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.162049    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:18.243470 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.243407    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:18.315823 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.315790    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:18.415578 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.415481    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:18.527937 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.527886    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:18.561687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.561662    2568 scope.go:117] "RemoveContainer" containerID="9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495"
Jul 27 19:26:18.561851 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:18.561830    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-459pm_openshift-console-operator(0bcb2391-77ca-4b27-854a-e63d209d7def)\"" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podUID="0bcb2391-77ca-4b27-854a-e63d209d7def"
Jul 27 19:26:18.626244 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:18.626208    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:21.154766 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:21.154689    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbvtm" event={"ID":"75395a11-454f-4e19-bd8d-842ccda55d1d","Type":"ContainerStarted","Data":"28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9"}
Jul 27 19:26:21.246112 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:21.246075    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:21.248421 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:21.248402    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0252630f-a0ee-4786-95ab-a6cf9dec8dec-metrics-certs\") pod \"network-metrics-daemon-s8jzv\" (UID: \"0252630f-a0ee-4786-95ab-a6cf9dec8dec\") " pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:21.493137 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:21.493068    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9v6qr\""
Jul 27 19:26:21.500891 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:21.500871    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:21.585059 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:21.584977    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:21.787220 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:21.787173    2568 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:26:21.787220 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02" Netns:"/var/run/netns/cab3abd0-a793-4806-8b32-168ba7e583fe" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:21.787220 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:21.787220 ip-10-0-143-3 kubenswrapper[2568]:  >
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:21.787259    2568 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02" Netns:"/var/run/netns/cab3abd0-a793-4806-8b32-168ba7e583fe" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:21.787284    2568 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02" Netns:"/var/run/netns/cab3abd0-a793-4806-8b32-168ba7e583fe" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:21.787436 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:21.787753 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:21.787376    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02\\\" Netns:\\\"/var/run/netns/cab3abd0-a793-4806-8b32-168ba7e583fe\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=f64eebeae13935b8b240e093c535f4133f9c45185d952a239527ec1979c9ff02;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.7.110:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:26:23.513890 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:23.513842    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:23.536304 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:23.536263    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:23.554160 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:23.554109    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:23.574575 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:23.574511    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:23.592207 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:23.592166    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:23Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:23.592207 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:23.592204    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:26:25.171941 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:25.171887    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mbvtm" event={"ID":"75395a11-454f-4e19-bd8d-842ccda55d1d","Type":"ContainerStarted","Data":"8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f"}
Jul 27 19:26:25.190313 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:25.190261    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:25.206836 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:25.206772    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:27.628289 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:27.628257    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:27.728495 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:27.728460    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:27.833616 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:27.833591    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:29.105935 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:29.105913    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:29.124031 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:29.123977    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:29.140836 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:29.140790    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:29.156482 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:29.156393    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:31.581245 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:31.581184    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:16Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:31.598083 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:31.598038    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:31.616097 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:31.616037    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:31.961610 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:31.961503    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-96d67c84-tfbsz" podUID="39ae04be-3837-4b32-b0ad-fa36f32fb045" containerName="registry" containerID="cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc" gracePeriod=30
Jul 27 19:26:32.194339 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.194306    2568 generic.go:356] "Generic (PLEG): container finished" podID="39ae04be-3837-4b32-b0ad-fa36f32fb045" containerID="bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc" exitCode=0
Jul 27 19:26:32.194590 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.194349    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-96d67c84-tfbsz" event={"ID":"39ae04be-3837-4b32-b0ad-fa36f32fb045","Type":"ContainerDied","Data":"bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc"}
Jul 27 19:26:32.210478 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.210454    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:32.227961 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.227875    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-trusted-ca\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.227961 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.227928    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.227959    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ae04be-3837-4b32-b0ad-fa36f32fb045-ca-trust-extracted\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228023    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-installation-pull-secrets\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228053    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-certificates\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228075    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-bound-sa-token\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228113    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7g6r\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-kube-api-access-d7g6r\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228346 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228155    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-image-registry-private-configuration\") pod \"39ae04be-3837-4b32-b0ad-fa36f32fb045\" (UID: \"39ae04be-3837-4b32-b0ad-fa36f32fb045\") "
Jul 27 19:26:32.228490 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228436    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:26:32.228645 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.228584    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:26:32.230043 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:32.229975    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:32.230894 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.230848    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:26:32.231032 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.231007    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-kube-api-access-d7g6r" (OuterVolumeSpecName: "kube-api-access-d7g6r") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "kube-api-access-d7g6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:26:32.231032 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.231013    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:26:32.231156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.231014    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:26:32.231156 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.231053    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:26:32.236819 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.236795    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ae04be-3837-4b32-b0ad-fa36f32fb045-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "39ae04be-3837-4b32-b0ad-fa36f32fb045" (UID: "39ae04be-3837-4b32-b0ad-fa36f32fb045"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:26:32.252187 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:32.252137    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:32.270615 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:32.270550    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:32.329163 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329133    2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-installation-pull-secrets\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329163 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329159    2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-certificates\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329163 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329169    2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-bound-sa-token\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329357 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329178    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7g6r\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-kube-api-access-d7g6r\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329357 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329188    2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ae04be-3837-4b32-b0ad-fa36f32fb045-image-registry-private-configuration\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329357 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329203    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ae04be-3837-4b32-b0ad-fa36f32fb045-trusted-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329357 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329214    2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ae04be-3837-4b32-b0ad-fa36f32fb045-registry-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:32.329357 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:32.329221    2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ae04be-3837-4b32-b0ad-fa36f32fb045-ca-trust-extracted\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:26:33.198454 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:33.198421    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-96d67c84-tfbsz" event={"ID":"39ae04be-3837-4b32-b0ad-fa36f32fb045","Type":"ContainerDied","Data":"0654ae2beae07ca8cc135990c2f3c65bd0d65e0d91d94c41b1e242f123f53ffa"}
Jul 27 19:26:33.198454 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:33.198454    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:33.198944 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:33.198477    2568 scope.go:117] "RemoveContainer" containerID="bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc"
Jul 27 19:26:33.222563 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.222496    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:33.245774 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.245727    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:33.285784 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.285717    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:33.316847 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.316792    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:33.349784 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.349731    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:33.368982 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.368920    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:33.566317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:33.566285    2568 scope.go:117] "RemoveContainer" containerID="9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495"
Jul 27 19:26:33.651874 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.651695    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:33.695610 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.695564    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:33.718764 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.718730    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:33.748601 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.748557    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:33.771801 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.771760    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:33Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:33.771801 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:33.771789    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:26:34.203846 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.203818    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:26:34.204290 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.203908    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" event={"ID":"0bcb2391-77ca-4b27-854a-e63d209d7def","Type":"ContainerStarted","Data":"f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06"}
Jul 27 19:26:34.204290 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.204204    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:34.225502 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.225430    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:34.246311 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.246264    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:34.270544 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.270470    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:34.292637 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.292489    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:34.523427 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.523391    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:34.541634 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.541555    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:34.571255 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.571179    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:34.591985 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.591919    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:34.618564 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.618463    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:34.717153 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.717120    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-pzckr"]
Jul 27 19:26:34.717430 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.717415    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ae04be-3837-4b32-b0ad-fa36f32fb045" containerName="registry"
Jul 27 19:26:34.717539 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.717434    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ae04be-3837-4b32-b0ad-fa36f32fb045" containerName="registry"
Jul 27 19:26:34.717539 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.717494    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ae04be-3837-4b32-b0ad-fa36f32fb045" containerName="registry"
Jul 27 19:26:34.720511 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.720490    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:34.725433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.725401    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 19:26:34.725433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.725423    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m2k46\""
Jul 27 19:26:34.725781 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.725765    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 19:26:34.740040 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.740016    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:34.743540 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.743486    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:34.746761 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.746742    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5p8\" (UniqueName: \"kubernetes.io/projected/edefbf77-cf8c-4499-abe5-65ee93dae38e-kube-api-access-qc5p8\") pod \"downloads-7c97dc4555-pzckr\" (UID: \"edefbf77-cf8c-4499-abe5-65ee93dae38e\") " pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:34.762972 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.762904    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:34.787631 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.787506    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:34.813158 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.813108    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:34.832853 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:34.832794    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:34.837536 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.837495    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:34.847723 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.847702    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5p8\" (UniqueName: \"kubernetes.io/projected/edefbf77-cf8c-4499-abe5-65ee93dae38e-kube-api-access-qc5p8\") pod \"downloads-7c97dc4555-pzckr\" (UID: \"edefbf77-cf8c-4499-abe5-65ee93dae38e\") " pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:34.859342 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:34.859323    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5p8\" (UniqueName: \"kubernetes.io/projected/edefbf77-cf8c-4499-abe5-65ee93dae38e-kube-api-access-qc5p8\") pod \"downloads-7c97dc4555-pzckr\" (UID: \"edefbf77-cf8c-4499-abe5-65ee93dae38e\") " pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:35.029440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:35.029400    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:35.228776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:35.228742    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:35.398853 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:35.398773    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:36.201655 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.201627    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:36.412354 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.412321    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:36.562314 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.562285    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:36.562656 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.562641    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:36.608938 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.608575    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:36.769952 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.769917    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:36.792333 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:36.792303    2568 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:26:36.792333 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d" Netns:"/var/run/netns/e93e7535-e84f-4ba8-8a50-28623ccf305a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:36.792333 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:36.792333 ip-10-0-143-3 kubenswrapper[2568]:  >
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:36.792359    2568 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d" Netns:"/var/run/netns/e93e7535-e84f-4ba8-8a50-28623ccf305a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:36.792378    2568 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d" Netns:"/var/run/netns/e93e7535-e84f-4ba8-8a50-28623ccf305a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:36.792481 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:36.792715 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:36.792442    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d\\\" Netns:\\\"/var/run/netns/e93e7535-e84f-4ba8-8a50-28623ccf305a\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=89f1a44b725b130a9a2fe62d50d0173de99657e3b91036a5d926dd5b96ab7f8d;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.7.110:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:26:36.867727 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:36.867655    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:39.174569 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:39.174535    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:39.273733 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:39.273699    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:39.603094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:39.603059    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:41.581959 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:41.581896    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:41.603223 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:41.603174    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:41.620137 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:41.620069    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:41.637810 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:41.637759    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:41.652217 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:41.652167    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:42.693898 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:42.693869    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:42.879168 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:42.879136    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:43.837297 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:43.837147    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:43.857392 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:43.857278    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:43.876388 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:43.876264    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:43.894133 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:43.894007    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:43.912097 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:43.911960    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:43Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:43.912097 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:43.912086    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:26:44.031323 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.031289    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f4f4f8d5b-n2pzh"]
Jul 27 19:26:44.082009 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.081977    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.085583 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.085556    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cg2fz\""
Jul 27 19:26:44.085718 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.085601    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 19:26:44.085718 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.085617    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 19:26:44.085718 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.085631    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 19:26:44.085924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.085904    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 19:26:44.085924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.085922    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 19:26:44.096655 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:44.096615    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:44.109239 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.109220    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-oauth-config\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.109322 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.109245    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-oauth-serving-cert\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.109322 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.109278    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-config\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.109405 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.109319    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-serving-cert\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.109405 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.109372    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-service-ca\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.109494 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.109424    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc5tc\" (UniqueName: \"kubernetes.io/projected/18eb1627-e40f-42f7-8cf9-cb58997bd08f-kube-api-access-fc5tc\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.116234 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:44.116186    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:44.136719 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:44.136665    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:44.152016 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:44.151937    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.169312 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:44.169245    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:44.185442 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:44.185393    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:44.209784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.209755    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-service-ca\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.209784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.209788    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc5tc\" (UniqueName: \"kubernetes.io/projected/18eb1627-e40f-42f7-8cf9-cb58997bd08f-kube-api-access-fc5tc\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.209987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.209832    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-oauth-config\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.209987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.209855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-oauth-serving-cert\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.209987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.209894    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-config\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.209987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.209920    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-serving-cert\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.210693 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.210561    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-service-ca\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.210693 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.210672    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-config\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.210921 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.210901    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-oauth-serving-cert\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.212550 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.212513    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-oauth-config\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.212904 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.212888    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-serving-cert\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.219318 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.219294    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc5tc\" (UniqueName: \"kubernetes.io/projected/18eb1627-e40f-42f7-8cf9-cb58997bd08f-kube-api-access-fc5tc\") pod \"console-f4f4f8d5b-n2pzh\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") " pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:44.391101 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:44.391028    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:48.827504 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:48.827471    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:48.938154 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:48.938128    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:50.561963 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:50.561923    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:50.562354 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:50.562274    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:50.843333    2568 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db" Netns:"/var/run/netns/0cf3a239-e6a8-4e98-88e2-e85cdec41706" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:  >
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:50.843390    2568 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db" Netns:"/var/run/netns/0cf3a239-e6a8-4e98-88e2-e85cdec41706" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:50.843407    2568 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db" Netns:"/var/run/netns/0cf3a239-e6a8-4e98-88e2-e85cdec41706" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:26:50.843917 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:50.843487    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db\\\" Netns:\\\"/var/run/netns/0cf3a239-e6a8-4e98-88e2-e85cdec41706\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=27c079a3257b27177d48977136b509650f801b04acd0c117803d5d8ba95c49db;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.7.110:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:26:51.578687 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:51.578633    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:51.592976 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:51.592926    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:51.611055 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:51.610999    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:51.629784 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:51.629733    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:51.644400 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:51.644353    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:51.660341 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:51.660274    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:52.611232 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.611200    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-777dc76db4-bxs67"]
Jul 27 19:26:52.731949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.731919    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.741121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.741095    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 19:26:52.754449 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.754402    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:26:52.772643 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-config\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.772759 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772661    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-serving-cert\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.772759 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772742    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-oauth-config\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.772849 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772771    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-oauth-serving-cert\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.772849 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772797    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsv4\" (UniqueName: \"kubernetes.io/projected/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-kube-api-access-mmsv4\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.772933 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772848    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-service-ca\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.772933 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.772866    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-trusted-ca-bundle\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.774257 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.774221    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:26:52.793385 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.793335    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca-bundle\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-mmsv4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:52Z\\\"}}\" for pod \"openshift-console\"/\"console-777dc76db4-bxs67\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.811048 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.811000    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:26:52.829946 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.829885    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:26:52.834746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.834724    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:52.845831 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.845791    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:26:52.862623 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:52.862543    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:26:52.873860 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873837    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-service-ca\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.873950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873862    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-trusted-ca-bundle\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.873950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873885    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-config\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.873950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-serving-cert\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.873950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873932    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-oauth-config\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.873950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873949    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-oauth-serving-cert\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.874159 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.873963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsv4\" (UniqueName: \"kubernetes.io/projected/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-kube-api-access-mmsv4\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.875010 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.874987    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-config\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.875130 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.874988    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-service-ca\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.879226 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.875198    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-trusted-ca-bundle\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.879226 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.875318    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-oauth-serving-cert\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.879226 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.877234    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-serving-cert\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.881390 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.881367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-oauth-config\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.885365 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.885344    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsv4\" (UniqueName: \"kubernetes.io/projected/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-kube-api-access-mmsv4\") pod \"console-777dc76db4-bxs67\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") " pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:52.933996 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:52.933966    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:53.040838 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:53.040799    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:26:54.269830 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:54.269720    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:54.288994 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:54.288886    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:54.306733 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:54.306619    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:54.335338 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:54.335231    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:54.385380 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:54.385252    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:26:54Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:26:54.385380 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:26:54.385373    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:26:57.695802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:57.695769    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:57.771535 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:57.771488    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:58.247507 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:58.247474    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:58.324209 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:58.324179    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:26:58.840318 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:26:58.840285    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:01.585109 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.585049    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:27:01.606767 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.606720    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:27:01.621651 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.621606    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:27:01.639415 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.639351    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:27:01.657140 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.657093    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:27:01.671314 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.671274    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:27:01.686990 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:01.686943    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca-bundle\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-mmsv4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:52Z\\\"}}\" for pod \"openshift-console\"/\"console-777dc76db4-bxs67\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:02.792371 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:02.792333    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:02.868349 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:02.868315    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:04.561964 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:04.561938    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:04.562319 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:04.562152    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:04.645148 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.644981    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:04.662853 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.662749    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:04.680549 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.680415    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:04.696644 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.696510    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:04.714170 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.714064    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:04Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:04.714170 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.714167    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:27:04.794935 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.794900    2568 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:27:04.794935 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8" Netns:"/var/run/netns/beece05f-7ac6-4ffd-a084-3a543daec32c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:27:04.794935 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:27:04.794935 ip-10-0-143-3 kubenswrapper[2568]:  >
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.794965    2568 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8" Netns:"/var/run/netns/beece05f-7ac6-4ffd-a084-3a543daec32c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.794983    2568 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8" Netns:"/var/run/netns/beece05f-7ac6-4ffd-a084-3a543daec32c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:04.795169 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:04.795047    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8\\\" Netns:\\\"/var/run/netns/beece05f-7ac6-4ffd-a084-3a543daec32c\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=fa1f3e78da9a5fe52c0a2699872497d4b2525c40c98f6fdf36c92a60ce4a93a8;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.7.110:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:27:04.997218 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:04.997131    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:05.098141 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:05.098110    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:05.197367 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:05.197332    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:07.525320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:07.525291    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:27:07.726189 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:07.726158    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/3.log"
Jul 27 19:27:08.925877 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:08.925848    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kkx66_a6852432-508f-4243-84ef-f9aebd2e2ae3/dns/0.log"
Jul 27 19:27:09.125182 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:09.125152    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kkx66_a6852432-508f-4243-84ef-f9aebd2e2ae3/kube-rbac-proxy/0.log"
Jul 27 19:27:10.125081 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:10.125049    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-slvtt_40900963-6214-4d92-8e58-7f7ab9fff63a/dns-node-resolver/0.log"
Jul 27 19:27:10.324981 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:10.324956    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69f478d788-xt6q7_dbdd8b60-768e-4803-92fa-c2bae4520a55/registry/0.log"
Jul 27 19:27:10.724340 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:10.724307    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\": container with ID starting with bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc not found: ID does not exist" containerID="bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc"
Jul 27 19:27:11.125080 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:11.125049    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcp6x_61ab69d2-8341-4561-8ed0-9873bff1eec1/node-ca/0.log"
Jul 27 19:27:11.525056 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:11.525028    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-sgscc_c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d/volume-data-source-validator/0.log"
Jul 27 19:27:11.582196 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.582137    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:27:11.600267 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.600211    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:27:11.615030 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.614984    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:27:11.631443 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.631356    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:27:11.647143 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.647094    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:27:11.660310 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.660263    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:27:11.674312 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:11.674266    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca-bundle\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-mmsv4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:52Z\\\"}}\" for pod \"openshift-console\"/\"console-777dc76db4-bxs67\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:12.924722 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:12.924687    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-jmfcc_bc5cba57-c9dc-4134-9de2-990c7a424e07/csi-driver/0.log"
Jul 27 19:27:13.125369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:13.125339    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-jmfcc_bc5cba57-c9dc-4134-9de2-990c7a424e07/csi-node-driver-registrar/0.log"
Jul 27 19:27:13.324373 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:13.324343    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-jmfcc_bc5cba57-c9dc-4134-9de2-990c7a424e07/csi-liveness-probe/0.log"
Jul 27 19:27:15.044208 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:15.044077    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:15.061938 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:15.061831    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:15.079059 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:15.078947    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:15.095416 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:15.095308    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:15.115355 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:15.115246    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:15Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:15.115355 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:15.115345    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:27:17.566186 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:17.566162    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:17.566583 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:17.566384    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:17.808589 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:17.808547    2568 log.go:32] "RunPodSandbox from runtime service failed" err=<
Jul 27 19:27:17.808589 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365" Netns:"/var/run/netns/3daa5e00-038c-416d-b9f6-eea168d83d18" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:27:17.808589 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:27:17.808589 ip-10-0-143-3 kubenswrapper[2568]:  >
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:17.808619    2568 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365" Netns:"/var/run/netns/3daa5e00-038c-416d-b9f6-eea168d83d18" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:17.808641    2568 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=<
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365" Netns:"/var/run/netns/3daa5e00-038c-416d-b9f6-eea168d83d18" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec" Path:"" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook "pod.network-node-identity.openshift.io": failed to call webhook: Post "https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s": dial tcp 172.30.7.110:9743: connect: connection refused
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]:  > pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:17.808834 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:17.808737    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"network-metrics-daemon-s8jzv_openshift-multus(0252630f-a0ee-4786-95ab-a6cf9dec8dec)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_network-metrics-daemon-s8jzv_openshift-multus_0252630f-a0ee-4786-95ab-a6cf9dec8dec_0(3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365): error adding pod openshift-multus_network-metrics-daemon-s8jzv to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365\\\" Netns:\\\"/var/run/netns/3daa5e00-038c-416d-b9f6-eea168d83d18\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-multus;K8S_POD_NAME=network-metrics-daemon-s8jzv;K8S_POD_INFRA_CONTAINER_ID=3a82bece625201e4b6427f8a120bf3d6136769b1aab88d68361e59e7386bb365;K8S_POD_UID=0252630f-a0ee-4786-95ab-a6cf9dec8dec\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-multus/network-metrics-daemon-s8jzv] networking: Multus: [openshift-multus/network-metrics-daemon-s8jzv/0252630f-a0ee-4786-95ab-a6cf9dec8dec]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: SetNetworkStatus: failed to update the pod network-metrics-daemon-s8jzv in out of cluster comm: status update failed for pod openshift-multus/network-metrics-daemon-s8jzv: Internal error occurred: failed calling webhook \\\"pod.network-node-identity.openshift.io\\\": failed to call webhook: Post \\\"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\\\": dial tcp 172.30.7.110:9743: connect: connection refused\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-multus/network-metrics-daemon-s8jzv" podUID="0252630f-a0ee-4786-95ab-a6cf9dec8dec"
Jul 27 19:27:21.581467 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.581380    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:27:21.597893 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.597837    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:27:21.612337 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.612285    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:27:21.631169 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.631113    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca-bundle\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-mmsv4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:52Z\\\"}}\" for pod \"openshift-console\"/\"console-777dc76db4-bxs67\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:21.650749 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.650688    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:27:21.671121 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.671077    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:27:21.686926 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:21.686881    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:27:25.245557 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:25.245430    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:25.266878 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:25.266773    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:25.283788 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:25.283683    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:25.305074 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:25.304959    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:25.320798 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:25.320675    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-27T19:27:25Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"images\\\":[{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16ca8b7c1184a105aac97293a35c94373262103579713746b69f4f125185b301\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c61b1bd630d745ebd63d45ede0bae739076b4213e12bea2e0bb124e8f4727ab0\\\"],\\\"sizeBytes\\\":1341984117},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:09a988dc79ac3bbc5aff0a5cae44cb8ae06b7d21a0c28d30499f6b5f63fcefd8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea7622cd6145f37fcac7b916bd4464831e2efdf19ebe0ec41c27047c814f23f0\\\"],\\\"sizeBytes\\\":1183199980},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0663b35dc117c67b903f052040d0ce01685d1e22781bc2ef837fb32d942f677c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89bbd9284fb92be98f07e3067b3dd3b08642713706cd6520e6ed5a7819c4d18e\\\"],\\\"sizeBytes\\\":876273280},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:73a0d41e53dabfc205782cf08f95c47e93f3857314598fbfd7b4d281b4a932b2\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a83451907191f154bed1dfc7f0174433f0075beb6b6b82b9ec61e07e402dc811\\\"],\\\"sizeBytes\\\":586731180},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f42f4c14e88b49108b37c533fbbc282996a50a648f57db40690d2b0e8d5c811\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e951c133c66e9b456835de5087842d6fc86fecfe5aad3ca560fdd9c719ed8ba\\\"],\\\"sizeBytes\\\":580608166},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:293fa839fb89fb9cbf1ce2af15b9a77fd74330235b9cfb7c52dd9036aeaa9460\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db45730f38384cb9f01ddcadeb44c84f374d2f7247d6bac02763ad686d826fbc\\\"],\\\"sizeBytes\\\":576024653},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c672072cc02663bcd473a2edc6c5142d36ddb28dea685e0cfc5d06f52e03d64\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b8fe761810ead4093b28f3e7ee9ef6b962fd3499250b1838a1b00cc77765d1bf\\\"],\\\"sizeBytes\\\":537939273},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:4a060b462bf74148809ea5f1bb39ace87d7a251fdb2d387f13508269115c0031\\\",\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\"],\\\"sizeBytes\\\":496206599},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b0cacd583df5a8b474e50ea6e4781dcc25c98a58c4003f8f4d2376d2e06d3e4\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7098150c7700e387c4b2b8a27c1f2f20e8a5cc53f34664fb04e53045726b32ac\\\"],\\\"sizeBytes\\\":456384232},{\\\"names\\\":[\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:f5b83335633379dc80042ac69074b3c2abde3d69f63b737c2de0dda6c7b7c77c\\\",\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\"],\\\"sizeBytes\\\":436864195},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8baa2658dad94f5ac6ec283c79202450a40a19107b35205491bdced10517aa\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b12683cf9c88863a8143e16207d8db92faef5df2d2e082879b78cc67d123c7f\\\"],\\\"sizeBytes\\\":386904411},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:787925288f5c2c026e0cbada3e6706c38be466bf3995c478d54c10defc389780\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e3ba88cacf3c747ed1559bdb550d4483a0e7a25ad4f76ef3c4588f72ae1d281e\\\"],\\\"sizeBytes\\\":349477060},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:233b9dfa42a292bdae9563a0cfb304f006bf4cf1f75229e9b3147c1019c74859\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf790441537ab71f6e39920d97984a14115bd52c94cb371ffd5cf4ef6899e344\\\"],\\\"sizeBytes\\\":347020814},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53008d8e30cdba8a421c865d7fd13497538e84d6843c1692df37914257440e91\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c6db0ab20c9e44aed5dff7711ce889fe6725a09b24499fc7971edcd307d1e942\\\"],\\\"sizeBytes\\\":343989083},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:953387b5083b2a3e349e8b745cabdfd7ffdfd16892f06e155060fb2cbb80cdff\\\"],\\\"sizeBytes\\\":336858231},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16e2a89862a2a359709d5e15ab45d48a41582b58f029f689a9703b468957426c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d23ad739611374dae5c04dc9c1be680bd178d811a2e617fbd8459cf73073df6e\\\"],\\\"sizeBytes\\\":318291126},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44dcc3802076a6fb1ea11dff5479cc5721d7c3452fb04af6023d965e5d92c987\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:924ddd9185690e32284b189b3d0c7f005f10029b713c963227fee4da98829559\\\"],\\\"sizeBytes\\\":317186379},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:22a20fce18eff6a339d1ccd64f4f1be550af86ecf079f3c9d9ee476e650ba013\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95fdde94b76e7c1c99d8115b18bd84e4a40bc75e55440c0a846806b4ed3ebbf0\\\"],\\\"sizeBytes\\\":315356876},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f01dca26528042bc3f05fdd53d5a72cb9f3bd70dcfe2f1d5eab626c7530dd859\\\"],\\\"sizeBytes\\\":313107438},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a779490844b723f8f9423a88c26299841ae072c25b69e4db9f94b761bea02b7\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca41189832d5c7e7b5399a4bdefc28be186945c77c50c799067c073e38d12481\\\"],\\\"sizeBytes\\\":307946540},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39fb880a46d245719ae227a8b8e803ed4945a53bbfd7aef02be61d262527989a\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46a09b61078082511645fb4b54f513d0b4c65ffe5abdd774765376c5b5df57e7\\\"],\\\"sizeBytes\\\":284044146},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:373a63186d156b4ce0ae2d8560e108441d17b09be34e6d6db2b3486bc43f7dfd\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79363f7abd34060a04015eceb060626d31d91d8c0aa51ce25ec751c237169fdc\\\"],\\\"sizeBytes\\\":278280461},{\\\"names\\\":[\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:aaa406e189db0611e77288dcd04e922263ad11392a640bf5edb93d6a527d4161\\\",\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\"],\\\"sizeBytes\\\":258221534},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:298798b82aa327b5d4845d2119544bf15d079f9273c26ae6c31b0ff785f7737c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:38be0cf7145e21615b72b12be031fd2071873afe8d11ac5b734b10e1e66fb357\\\"],\\\"sizeBytes\\\":250064697},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e863246db3b0816c2d1279b3d1a2ee5606624d3e1c7601c715ddc7309e7b68c\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f984df1cb2384583cc1b1d74cc304af86ed19d269145c54f380ee1b6b11ebeff\\\"],\\\"sizeBytes\\\":235560681},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d1d940b941cc3129d8c30028e2371b5741771afc1860c872c66cdc8ebda6c0fe\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0fbdb86532d771d5307b5972b2013a378dfa3a213b2f5960eece8410c770ca3\\\"],\\\"sizeBytes\\\":235302136},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15259073275e2789accc887bd84daeb72a005385030ffcfd81af0ece73a97191\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:15ab63548751bddcfa77c870dddc6d9bb822b23f82cba324e5f1747c0d35abe8\\\"],\\\"sizeBytes\\\":222771689},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:062f884217d33f8982f101f5b2bee51590a578164f2707db799b0dce80c399e8\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64a796986118e9bcb9f32dd15fba29e2aa5906f3068c8209a3ad21b66cd7fe06\\\"],\\\"sizeBytes\\\":218369365},{\\\"names\\\":[\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb\\\",\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7fef1a9d7fb1e554165cc9ff6b4ca01027ee426c0e28b7a9f81ce5339b8bb53f\\\"],\\\"sizeBytes\\\":206446249},{\\\"names\\\":[\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:6270079008ddaa493b09474324185a9224c3662bd93826c447f03f09cdd49a72\\\",\\\"registry.redhat.io/multicluster-engine/managed-serviceaccount-rhel9@sha256:7a4f13a9159e8b4f596dd708822553dc76633f2d93ac9ddc0ef7652aeb0bc824\\\"],\\\"sizeBytes\\\":190496460}]}}\" for node \"ip-10-0-143-3.ec2.internal\": Internal error occurred: failed calling webhook \"node.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/node?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused"
Jul 27 19:27:25.320798 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:25.320797    2568 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Jul 27 19:27:31.578867 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.578808    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bcb2391-77ca-4b27-854a-e63d209d7def\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://f3703f614f32cd5b287049e28e6721e361ae8aa9ef70b9484b2d761eb1f97c06\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20ad3393e0e7304db066dbe287db5ab44be9032217e1e5c94406d3e3bd1fe672\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://9605396ee658a53fc3fc5cdf285064f42bc38a109520ea669d97223db562f495\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-07-27T19:26:00Z\\\",\\\"message\\\":\\\"W0727 19:26:00.652956       1 cmd.go:257] Using insecure, self-signed certificates\\\\nF0727 19:26:00.653062       1 cmd.go:182] mkdir /tmp/serving-cert-3054102461: read-only file system\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-27T19:26:00Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:33Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-t47pj\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-584dcb6b54-459pm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console-operator/console-operator-584dcb6b54-459pm"
Jul 27 19:27:31.593886 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.593845    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"edefbf77-cf8c-4499-abe5-65ee93dae38e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:34Z\\\",\\\"message\\\":\\\"containers with unready status: [download-server]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffac3fc7b593f418d0c90982f999bcbcfab19d67577a89bfd630d5f1a4831d6e\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"download-server\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5p8\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:34Z\\\"}}\" for pod \"openshift-console\"/\"downloads-7c97dc4555-pzckr\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:27:31.609112 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.609058    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:52Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca-bundle\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-mmsv4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:52Z\\\"}}\" for pod \"openshift-console\"/\"console-777dc76db4-bxs67\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:31.628048 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.627995    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"39ae04be-3837-4b32-b0ad-fa36f32fb045\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:32Z\\\",\\\"message\\\":null,\\\"reason\\\":\\\"PodCompleted\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://bb42635d9f02b1d2a177b5f353f5a79f7ad2a09d6feb33c42c1bd7c013e504bc\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-07-27T19:26:31Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-07-27T19:25:52Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-d7g6r\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Succeeded\\\"}}\" for pod \"openshift-image-registry\"/\"image-registry-96d67c84-tfbsz\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-96d67c84-tfbsz"
Jul 27 19:27:31.646388 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.646335    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dbdd8b60-768e-4803-92fa-c2bae4520a55\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:29Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"},\\\"containerID\\\":\\\"cri-o://037e01b06993c8d44926d51f074169147bc0f635c2866da10ca6ccb0b0cd4626\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"registry\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"256Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:07Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000300000],\\\"uid\\\":1000300000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/cloud\\\",\\\"name\\\":\\\"image-registry-private-configuration\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/etc/secrets\\\",\\\"name\\\":\\\"registry-tls\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted\\\",\\\"name\\\":\\\"ca-trust-extracted\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/source/anchors\\\",\\\"name\\\":\\\"registry-certificates\\\"},{\\\"mountPath\\\":\\\"/usr/share/pki/ca-trust-source\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/lib/kubelet/\\\",\\\"name\\\":\\\"installation-pull-secrets\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-p9lk4\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-image-registry\"/\"image-registry-69f478d788-xt6q7\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-image-registry/image-registry-69f478d788-xt6q7"
Jul 27 19:27:31.661425 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.661379    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"18eb1627-e40f-42f7-8cf9-cb58997bd08f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"observedGeneration\\\":1,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Initialized\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastProbeTime\\\":null,\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:44Z\\\",\\\"message\\\":\\\"containers with unready status: [console]\\\",\\\"observedGeneration\\\":1,\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":1,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"containerStatuses\\\":[{\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:144d362a755acd381a5dd1a7caf6b134f24de4b94fa2c5c98a0639382a9efeca\\\",\\\"imageID\\\":\\\"\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"console\\\",\\\"ready\\\":false,\\\"restartCount\\\":0,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"reason\\\":\\\"ContainerCreating\\\"}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/serving-cert\\\",\\\"name\\\":\\\"console-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-config\\\",\\\"name\\\":\\\"console-oauth-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/console-config\\\",\\\"name\\\":\\\"console-config\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/service-ca\\\",\\\"name\\\":\\\"service-ca\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/oauth-serving-cert\\\",\\\"name\\\":\\\"oauth-serving-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-fc5tc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"hostIP\\\":\\\"10.0.143.3\\\",\\\"hostIPs\\\":[{\\\"ip\\\":\\\"10.0.143.3\\\"}],\\\"observedGeneration\\\":1,\\\"startTime\\\":\\\"2026-07-27T19:26:44Z\\\"}}\" for pod \"openshift-console\"/\"console-f4f4f8d5b-n2pzh\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:27:31.679731 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:31.679667    2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"75395a11-454f-4e19-bd8d-842ccda55d1d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-27T19:26:25Z\\\",\\\"message\\\":null,\\\"reason\\\":null,\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://28645159046dac25af8f0f83c364891bb4873cc9bb27779f503b32a7b28e75c9\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-exporter@sha256:49ac9e2d3ec12a66e92f72969141d01c947d7918fa193c0e54876dfee873711d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"exporter\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://8442d0ba90265054bc017b0b403fe9454e196f6f22e53c07780e7a537954e60f\\\",\\\"image\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-insights-runtime-extractor@sha256:49ed43f89842b75a35ac5fb95262a12f5eb9408a2080b06a34665afe9c21b3d7\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"extractor\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:24Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0],\\\"uid\\\":0}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/crio.sock\\\",\\\"name\\\":\\\"crio-socket\\\"},{\\\"mountPath\\\":\\\"/data\\\",\\\"name\\\":\\\"data-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]},{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://d207fc11b1ffae2d7ead2eeed9e46746807804702b2ddaa16199eb9384e2df2b\\\",\\\"image\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy:latest\\\",\\\"imageID\\\":\\\"quay.io/openshift/origin-kube-rbac-proxy@sha256:955a4c242f8bb3cdfdf2c43e8c852cdd1689a1c76437f76967f5b23c3382126d\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"kube-rbac-proxy\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-27T19:26:17Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":65534,\\\"supplementalGroups\\\":[65534],\\\"uid\\\":65534}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/etc/tls/private\\\",\\\"name\\\":\\\"insights-runtime-extractor-tls\\\"},{\\\"mountPath\\\":\\\"/etc/kube-rbac-proxy\\\",\\\"name\\\":\\\"kube-rbac-proxy-cm\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-wrc2x\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.13\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.13\\\"}]}}\" for pod \"openshift-insights\"/\"insights-runtime-extractor-mbvtm\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-69888ee1-6e4e-49ad-86b3-871be4aaf8b0.svc:9743/pod?timeout=10s\": dial tcp 172.30.7.110:9743: connect: connection refused" pod="openshift-insights/insights-runtime-extractor-mbvtm"
Jul 27 19:27:32.561425 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:32.561390    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:32.561711 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:32.561699    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8jzv"
Jul 27 19:27:32.691320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:32.691290    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:32.701931 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:32.701900    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s8jzv"]
Jul 27 19:27:32.705511 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:32.705482    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252630f_a0ee_4786_95ab_a6cf9dec8dec.slice/crio-18cbb80651e677568fc48903422e5fd21635181cb683af5efae1cc7e85a7f8e1 WatchSource:0}: Error finding container 18cbb80651e677568fc48903422e5fd21635181cb683af5efae1cc7e85a7f8e1: Status 404 returned error can't find the container with id 18cbb80651e677568fc48903422e5fd21635181cb683af5efae1cc7e85a7f8e1
Jul 27 19:27:32.790660 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:32.790609    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:33.386278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:33.386240    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s8jzv" event={"ID":"0252630f-a0ee-4786-95ab-a6cf9dec8dec","Type":"ContainerStarted","Data":"18cbb80651e677568fc48903422e5fd21635181cb683af5efae1cc7e85a7f8e1"}
Jul 27 19:27:34.391559 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.391502    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s8jzv" event={"ID":"0252630f-a0ee-4786-95ab-a6cf9dec8dec","Type":"ContainerStarted","Data":"c6631ba6a82d200db6d332ce0e689bcfcd4fd952b9483f59b626d64972f7340b"}
Jul 27 19:27:34.392035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.391583    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s8jzv" event={"ID":"0252630f-a0ee-4786-95ab-a6cf9dec8dec","Type":"ContainerStarted","Data":"a396064f9e97cf43267bbb58b00771ce7b7bfe1b010c1d69f4386ecdf081ebac"}
Jul 27 19:27:34.415045 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.414996    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-459pm" podStartSLOduration=116.053734412 podStartE2EDuration="1m58.414980488s" totalImagesPullingTime="2.361246076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:34.414159854 +0000 UTC m=+203.410007831" watchObservedRunningTime="2026-07-27 19:27:34.414980488 +0000 UTC m=+203.410828512"
Jul 27 19:27:34.451262 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.451228    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-pzckr"]
Jul 27 19:27:34.480283 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.480256    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777dc76db4-bxs67"]
Jul 27 19:27:34.496428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.496399    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-96d67c84-tfbsz"]
Jul 27 19:27:34.502505 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.502478    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-96d67c84-tfbsz"]
Jul 27 19:27:34.579476 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.579436    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-pzckr"]
Jul 27 19:27:34.582603 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:34.582565    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedefbf77_cf8c_4499_abe5_65ee93dae38e.slice/crio-1a9db525eaa3fd0329f4962810ac604a0678fa3145188343af89970faf5cb06f WatchSource:0}: Error finding container 1a9db525eaa3fd0329f4962810ac604a0678fa3145188343af89970faf5cb06f: Status 404 returned error can't find the container with id 1a9db525eaa3fd0329f4962810ac604a0678fa3145188343af89970faf5cb06f
Jul 27 19:27:34.584226 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.584200    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4f4f8d5b-n2pzh"]
Jul 27 19:27:34.589289 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.589234    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s8jzv" podStartSLOduration=201.52061159 podStartE2EDuration="3m22.589219124s" totalImagesPullingTime="1.068607534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:34.588241505 +0000 UTC m=+203.584089484" watchObservedRunningTime="2026-07-27 19:27:34.589219124 +0000 UTC m=+203.585067102"
Jul 27 19:27:34.618383 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.618339    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mbvtm" podStartSLOduration=102.914722529 podStartE2EDuration="1m54.618319361s" totalImagesPullingTime="11.703596832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:25:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:34.618010147 +0000 UTC m=+203.613858124" watchObservedRunningTime="2026-07-27 19:27:34.618319361 +0000 UTC m=+203.614167339"
Jul 27 19:27:34.710015 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.709984    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4f4f8d5b-n2pzh"]
Jul 27 19:27:34.713183 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:34.713156    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18eb1627_e40f_42f7_8cf9_cb58997bd08f.slice/crio-31fef50d1d36275b6af92156797f04e954b55f295c3f002fd0629a31bcd7b213 WatchSource:0}: Error finding container 31fef50d1d36275b6af92156797f04e954b55f295c3f002fd0629a31bcd7b213: Status 404 returned error can't find the container with id 31fef50d1d36275b6af92156797f04e954b55f295c3f002fd0629a31bcd7b213
Jul 27 19:27:34.715207 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.715112    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:34.743199 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.743174    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777dc76db4-bxs67"]
Jul 27 19:27:34.746921 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:34.746891    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6271cc_a4dc_4a3d_98bd_3036e6f248db.slice/crio-bd80867b90bf56d6c2e20fa994d2da27f736ca0c2f96e73f835205218502bcad WatchSource:0}: Error finding container bd80867b90bf56d6c2e20fa994d2da27f736ca0c2f96e73f835205218502bcad: Status 404 returned error can't find the container with id bd80867b90bf56d6c2e20fa994d2da27f736ca0c2f96e73f835205218502bcad
Jul 27 19:27:34.794036 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.794004    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:34.862289 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:34.862256    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:35.019386 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.019361    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:35.120830 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.120793    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:35.221746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.221695    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:35.403044 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.402959    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777dc76db4-bxs67" event={"ID":"fa6271cc-a4dc-4a3d-98bd-3036e6f248db","Type":"ContainerStarted","Data":"bd80867b90bf56d6c2e20fa994d2da27f736ca0c2f96e73f835205218502bcad"}
Jul 27 19:27:35.407403 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.407370    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-pzckr" event={"ID":"edefbf77-cf8c-4499-abe5-65ee93dae38e","Type":"ContainerStarted","Data":"1a9db525eaa3fd0329f4962810ac604a0678fa3145188343af89970faf5cb06f"}
Jul 27 19:27:35.413135 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.413111    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4f4f8d5b-n2pzh" event={"ID":"18eb1627-e40f-42f7-8cf9-cb58997bd08f","Type":"ContainerStarted","Data":"31fef50d1d36275b6af92156797f04e954b55f295c3f002fd0629a31bcd7b213"}
Jul 27 19:27:35.573483 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:35.573446    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ae04be-3837-4b32-b0ad-fa36f32fb045" path="/var/lib/kubelet/pods/39ae04be-3837-4b32-b0ad-fa36f32fb045/volumes"
Jul 27 19:27:39.433014 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:39.432978    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777dc76db4-bxs67" event={"ID":"fa6271cc-a4dc-4a3d-98bd-3036e6f248db","Type":"ContainerStarted","Data":"08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3"}
Jul 27 19:27:39.434883 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:39.434850    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4f4f8d5b-n2pzh" event={"ID":"18eb1627-e40f-42f7-8cf9-cb58997bd08f","Type":"ContainerStarted","Data":"1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961"}
Jul 27 19:27:39.459697 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:39.458133    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-777dc76db4-bxs67" podStartSLOduration=43.615068556 podStartE2EDuration="47.458113586s" totalImagesPullingTime="3.84304503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:39.45560841 +0000 UTC m=+208.451456384" watchObservedRunningTime="2026-07-27 19:27:39.458113586 +0000 UTC m=+208.453961564"
Jul 27 19:27:39.478024 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:39.477964    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f4f4f8d5b-n2pzh" podStartSLOduration=51.590341164 podStartE2EDuration="55.477927482s" totalImagesPullingTime="3.887586318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:39.476717549 +0000 UTC m=+208.472565531" watchObservedRunningTime="2026-07-27 19:27:39.477927482 +0000 UTC m=+208.473775516"
Jul 27 19:27:42.407119 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:42.407088    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:43.041301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.041263    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:43.041489 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.041337    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:43.046997 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.046973    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:43.456490 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.456417    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:27:43.519023 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.518988    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4f4f8d5b-n2pzh"]
Jul 27 19:27:43.628649 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.628621    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:43.736925 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.736837    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:43.845669 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:43.845639    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.071393 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.071366    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.199163 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.199135    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.268783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.268180    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-77m7m"]
Jul 27 19:27:44.273751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.273725    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.276758 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.276729    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 19:27:44.276911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.276812    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 19:27:44.278850 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.278825    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 19:27:44.278957 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.278925    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 19:27:44.279072 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.279052    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6nkqs\""
Jul 27 19:27:44.327229 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.327127    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.359868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.359829    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-wtmp\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.359870    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.359928    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-textfile\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.359967    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-sys\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.359985    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-tls\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.360038    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.360096    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs72l\" (UniqueName: \"kubernetes.io/projected/4c30b188-012b-4ed4-8908-768f6fbb7e2a-kube-api-access-vs72l\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.360167    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-root\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.360205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.360195    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c30b188-012b-4ed4-8908-768f6fbb7e2a-metrics-client-ca\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.391958 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.391926    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:27:44.418083 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.418046    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.461456 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs72l\" (UniqueName: \"kubernetes.io/projected/4c30b188-012b-4ed4-8908-768f6fbb7e2a-kube-api-access-vs72l\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461456 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461462    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-root\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c30b188-012b-4ed4-8908-768f6fbb7e2a-metrics-client-ca\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461498    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-wtmp\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461517    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461564    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-root\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461650    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-textfile\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461701    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-sys\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-wtmp\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461726    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-tls\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461773    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461781    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c30b188-012b-4ed4-8908-768f6fbb7e2a-sys\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:44.461804    2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:27:44.461878    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-tls podName:4c30b188-012b-4ed4-8908-768f6fbb7e2a nodeName:}" failed. No retries permitted until 2026-07-27 19:27:44.961859066 +0000 UTC m=+213.957707021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-tls") pod "node-exporter-77m7m" (UID: "4c30b188-012b-4ed4-8908-768f6fbb7e2a") : secret "node-exporter-tls" not found
Jul 27 19:27:44.461989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.461935    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-textfile\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.462473 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.462135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-accelerators-collector-config\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.462473 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.462189    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c30b188-012b-4ed4-8908-768f6fbb7e2a-metrics-client-ca\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.464077 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.464060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.471211 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.471190    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs72l\" (UniqueName: \"kubernetes.io/projected/4c30b188-012b-4ed4-8908-768f6fbb7e2a-kube-api-access-vs72l\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.529207 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.529168    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.619142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.619030    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.719517 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.719480    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:44.967268 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.967181    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-tls\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:44.969867 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:44.969846    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c30b188-012b-4ed4-8908-768f6fbb7e2a-node-exporter-tls\") pod \"node-exporter-77m7m\" (UID: \"4c30b188-012b-4ed4-8908-768f6fbb7e2a\") " pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:45.090737 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:45.090694    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:45.184058 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:45.184018    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77m7m"
Jul 27 19:27:45.197755 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:45.197717    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c30b188_012b_4ed4_8908_768f6fbb7e2a.slice/crio-c3a61c8078acec2fefa771183f12466d9a56cc5971005c0a2c3b00e2e2b06c2e WatchSource:0}: Error finding container c3a61c8078acec2fefa771183f12466d9a56cc5971005c0a2c3b00e2e2b06c2e: Status 404 returned error can't find the container with id c3a61c8078acec2fefa771183f12466d9a56cc5971005c0a2c3b00e2e2b06c2e
Jul 27 19:27:45.228255 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:45.228177    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:45.458987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:45.458953    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77m7m" event={"ID":"4c30b188-012b-4ed4-8908-768f6fbb7e2a","Type":"ContainerStarted","Data":"c3a61c8078acec2fefa771183f12466d9a56cc5971005c0a2c3b00e2e2b06c2e"}
Jul 27 19:27:45.526891 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:45.526856    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:46.464953 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:46.464855    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77m7m" event={"ID":"4c30b188-012b-4ed4-8908-768f6fbb7e2a","Type":"ContainerStarted","Data":"1f88193f82cb1d909408feebe7bf0173f9555826064b0cbd1dc2aab2205eab02"}
Jul 27 19:27:47.198003 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.197846    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:47.200199 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.200174    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-849d4f79c7-rgckd"]
Jul 27 19:27:47.205111 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.205092    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.209188 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.209165    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 19:27:47.209308 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.209200    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 19:27:47.209507 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.209464    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 19:27:47.209608 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.209581    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 19:27:47.209687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.209667    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 19:27:47.210759 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.210740    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6h2plkjm8l2vn\""
Jul 27 19:27:47.210874 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.210780    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4zptl\""
Jul 27 19:27:47.227430 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.227404    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-849d4f79c7-rgckd"]
Jul 27 19:27:47.291970 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.291929    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292161 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.291981    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-tls\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292161 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.292058    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292161 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.292124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292325 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.292174    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-grpc-tls\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292325 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.292243    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12a874c-f838-42cd-98db-2936008d6f86-metrics-client-ca\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292425 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.292324    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.292425 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.292352    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2557n\" (UniqueName: \"kubernetes.io/projected/e12a874c-f838-42cd-98db-2936008d6f86-kube-api-access-2557n\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.315535 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.315492    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:47.392986 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.392953    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.392986 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.392991    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-tls\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.393231 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.393020    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.393231 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.393181    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.393312 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.393226    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-grpc-tls\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.393312 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.393285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12a874c-f838-42cd-98db-2936008d6f86-metrics-client-ca\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.393402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.393323    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.393402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.393356    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2557n\" (UniqueName: \"kubernetes.io/projected/e12a874c-f838-42cd-98db-2936008d6f86-kube-api-access-2557n\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.394122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.394075    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e12a874c-f838-42cd-98db-2936008d6f86-metrics-client-ca\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.396283 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.396239    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.396283 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.396259    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.396943 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.396921    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-grpc-tls\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.397118 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.397099    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.397252 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.397194    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.397642 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.397624    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e12a874c-f838-42cd-98db-2936008d6f86-secret-thanos-querier-tls\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.406268 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.406170    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2557n\" (UniqueName: \"kubernetes.io/projected/e12a874c-f838-42cd-98db-2936008d6f86-kube-api-access-2557n\") pod \"thanos-querier-849d4f79c7-rgckd\" (UID: \"e12a874c-f838-42cd-98db-2936008d6f86\") " pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.516057 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.515977    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:27:47.714501 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.714457    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:47.814738 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:47.814700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:48.855904 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:48.855826    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:48.870440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:48.870411    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5479c7dc6-kp49j"]
Jul 27 19:27:48.874279 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:48.874254    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:48.893133 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:48.893107    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5479c7dc6-kp49j"]
Jul 27 19:27:48.939908 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:48.939884    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:49.008241 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008188    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-console-config\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.008433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc76w\" (UniqueName: \"kubernetes.io/projected/deed3143-d011-4e9b-843d-ca0b73042493-kube-api-access-zc76w\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.008433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-oauth-config\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.008433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008414    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-service-ca\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.008627 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008464    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-trusted-ca-bundle\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.008627 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008508    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-serving-cert\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.008627 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.008582    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-oauth-serving-cert\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.025792 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.025757    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:49.102900 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.102863    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:49.109623 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109552    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-serving-cert\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.109623 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109609    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-oauth-serving-cert\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.109813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109639    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-console-config\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.109813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109668    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc76w\" (UniqueName: \"kubernetes.io/projected/deed3143-d011-4e9b-843d-ca0b73042493-kube-api-access-zc76w\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.109813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109715    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-oauth-config\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.109813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-service-ca\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.109813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.109770    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-trusted-ca-bundle\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.110477 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.110446    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-oauth-serving-cert\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.110734 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.110711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-service-ca\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.110804 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.110761    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-trusted-ca-bundle\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.110804 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.110761    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-console-config\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.112431 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.112411    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-serving-cert\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.112761 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.112737    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-oauth-config\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.123578 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.123557    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc76w\" (UniqueName: \"kubernetes.io/projected/deed3143-d011-4e9b-843d-ca0b73042493-kube-api-access-zc76w\") pod \"console-5479c7dc6-kp49j\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") " pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.185739 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.185694    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:27:49.211114 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:49.211078    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:50.469473 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.469433    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:50.488642 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.488611    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:27:50.492587 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.492561    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.496341 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.496314    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 19:27:50.496341 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.496324    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 19:27:50.496570 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.496341    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hblm8\""
Jul 27 19:27:50.496763 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.496697    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 19:27:50.496922 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.496901    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 19:27:50.497054 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.497035    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 19:27:50.497462 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.497430    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 19:27:50.497462 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.497449    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 19:27:50.497462 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.497433    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 19:27:50.497671 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.497435    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 19:27:50.498848 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.498829    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6a28c1eqm97nm\""
Jul 27 19:27:50.498941 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.498877    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 19:27:50.498941 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.498876    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 19:27:50.499313 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.499294    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 19:27:50.502051 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.502034    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 19:27:50.502328 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.502106    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 19:27:50.514772 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.514753    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 19:27:50.532065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.532038    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:27:50.616094 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.616056    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:50.620722 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.620866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620738    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.620866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620774    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.620866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620793    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.620866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621027 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620879    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621027 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620911    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621027 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620934    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621027 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621027 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.620993    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621033    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621054    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621079    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621416 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621416 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621297    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621416 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621325    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621416 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621352    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbflw\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-kube-api-access-hbflw\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.621616 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.621438    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.702057 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.702026    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:50.722395 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722326    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722395 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722421    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbflw\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-kube-api-access-hbflw\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722602    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722634    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722688    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722738    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722776    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722827    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722851    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.722911 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722905    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.723320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722930    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.723320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722955    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.723320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.722987    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.723320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.723096    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.723512 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.723343    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.723597 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.723544    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.724628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.724285    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.724628 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.724328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.725127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.725100    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.726367 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.726342    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.727029 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.726994    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.727950 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.727925    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.728057 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.728033    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.728150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.728036    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.728150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.728082    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.728300 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.728281    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.729836 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.729771    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.729836 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.729773    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.729836 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.729815    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.730145 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.730072    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.730695 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.730659    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.730806 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.730786    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.734535 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.734499    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbflw\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-kube-api-access-hbflw\") pod \"prometheus-k8s-0\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:50.804323 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:50.804288    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:27:53.236189 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.236161    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5479c7dc6-kp49j"]
Jul 27 19:27:53.238403 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:53.238373    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeed3143_d011_4e9b_843d_ca0b73042493.slice/crio-916e97d777b7b24dd9ecb2e5626f7839eb7a1137ef489568e6a1cdf154c4515f WatchSource:0}: Error finding container 916e97d777b7b24dd9ecb2e5626f7839eb7a1137ef489568e6a1cdf154c4515f: Status 404 returned error can't find the container with id 916e97d777b7b24dd9ecb2e5626f7839eb7a1137ef489568e6a1cdf154c4515f
Jul 27 19:27:53.469938 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.469862    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-849d4f79c7-rgckd"]
Jul 27 19:27:53.471993 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:53.471957    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12a874c_f838_42cd_98db_2936008d6f86.slice/crio-12f3fc7fb2e3400ef4b9e06946cd986c947e52bba833300c9ea560b9760d05c9 WatchSource:0}: Error finding container 12f3fc7fb2e3400ef4b9e06946cd986c947e52bba833300c9ea560b9760d05c9: Status 404 returned error can't find the container with id 12f3fc7fb2e3400ef4b9e06946cd986c947e52bba833300c9ea560b9760d05c9
Jul 27 19:27:53.477343 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.477317    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:27:53.479017 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:53.478990    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01ea71c_c5d0_417c_8b48_744d3f6b75a8.slice/crio-26af8fce4aaaa4cd661b316d12d6780f0b0bcef5dc4a095ec8f6ba346ed0fdc8 WatchSource:0}: Error finding container 26af8fce4aaaa4cd661b316d12d6780f0b0bcef5dc4a095ec8f6ba346ed0fdc8: Status 404 returned error can't find the container with id 26af8fce4aaaa4cd661b316d12d6780f0b0bcef5dc4a095ec8f6ba346ed0fdc8
Jul 27 19:27:53.492852 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.492820    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"12f3fc7fb2e3400ef4b9e06946cd986c947e52bba833300c9ea560b9760d05c9"}
Jul 27 19:27:53.494674 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.494641    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5479c7dc6-kp49j" event={"ID":"deed3143-d011-4e9b-843d-ca0b73042493","Type":"ContainerStarted","Data":"bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327"}
Jul 27 19:27:53.494841 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.494824    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5479c7dc6-kp49j" event={"ID":"deed3143-d011-4e9b-843d-ca0b73042493","Type":"ContainerStarted","Data":"916e97d777b7b24dd9ecb2e5626f7839eb7a1137ef489568e6a1cdf154c4515f"}
Jul 27 19:27:53.496557 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.496510    2568 generic.go:356] "Generic (PLEG): container finished" podID="4c30b188-012b-4ed4-8908-768f6fbb7e2a" containerID="1f88193f82cb1d909408feebe7bf0173f9555826064b0cbd1dc2aab2205eab02" exitCode=0
Jul 27 19:27:53.496650 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.496560    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77m7m" event={"ID":"4c30b188-012b-4ed4-8908-768f6fbb7e2a","Type":"ContainerDied","Data":"1f88193f82cb1d909408feebe7bf0173f9555826064b0cbd1dc2aab2205eab02"}
Jul 27 19:27:53.498465 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.498431    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-pzckr" event={"ID":"edefbf77-cf8c-4499-abe5-65ee93dae38e","Type":"ContainerStarted","Data":"97b00e65d3f655121e9670bbdb5c6eaec7e73a96deedcbfdae9ae987309d8637"}
Jul 27 19:27:53.499914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.499058    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:27:53.501702 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.501686    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"26af8fce4aaaa4cd661b316d12d6780f0b0bcef5dc4a095ec8f6ba346ed0fdc8"}
Jul 27 19:27:53.511153 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.511135    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-pzckr"
Jul 27 19:27:53.517853 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.516798    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5479c7dc6-kp49j" podStartSLOduration=5.516778362 podStartE2EDuration="5.516778362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:53.516589086 +0000 UTC m=+222.512437249" watchObservedRunningTime="2026-07-27 19:27:53.516778362 +0000 UTC m=+222.512626340"
Jul 27 19:27:53.540648 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:53.540601    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-pzckr" podStartSLOduration=60.897483459 podStartE2EDuration="1m19.540584178s" totalImagesPullingTime="18.643100719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:53.53946081 +0000 UTC m=+222.535308788" watchObservedRunningTime="2026-07-27 19:27:53.540584178 +0000 UTC m=+222.536432156"
Jul 27 19:27:54.479104 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.478447    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5479c7dc6-kp49j"]
Jul 27 19:27:54.512547 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.512489    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77m7m" event={"ID":"4c30b188-012b-4ed4-8908-768f6fbb7e2a","Type":"ContainerStarted","Data":"b50864fe1d904a8b3fcd067ff68c6065a824d8ee5d479a2afe3c196d1382b974"}
Jul 27 19:27:54.512978 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.512955    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77m7m" event={"ID":"4c30b188-012b-4ed4-8908-768f6fbb7e2a","Type":"ContainerStarted","Data":"27ab8a59c58c3a7516ff24b6769b87d6f2453288780b938c1169429d2fbe7cc5"}
Jul 27 19:27:54.525746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.524306    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c5d7d864c-q6vlv"]
Jul 27 19:27:54.527504 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.527469    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.543963 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.543886    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c5d7d864c-q6vlv"]
Jul 27 19:27:54.546273 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.545567    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-77m7m" podStartSLOduration=9.571543371 podStartE2EDuration="10.545517847s" totalImagesPullingTime="897.228156ms" totalInitContainerRuntime="76.74632ms" isStatefulPod=true podCreationTimestamp="2026-07-27 19:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:54.543861862 +0000 UTC m=+223.539709837" watchObservedRunningTime="2026-07-27 19:27:54.545517847 +0000 UTC m=+223.541365823"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8p7\" (UniqueName: \"kubernetes.io/projected/78861148-278d-46ef-a9d8-cc188609acc8-kube-api-access-5j8p7\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-serving-cert\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-service-ca\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555504    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-oauth-config\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555576    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-trusted-ca-bundle\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555620    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-console-config\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.557551 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.555699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-oauth-serving-cert\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656265    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-oauth-config\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-trusted-ca-bundle\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656354    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-console-config\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656709 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656401    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-oauth-serving-cert\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656709 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656459    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8p7\" (UniqueName: \"kubernetes.io/projected/78861148-278d-46ef-a9d8-cc188609acc8-kube-api-access-5j8p7\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656709 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656488    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-serving-cert\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.656709 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.656518    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-service-ca\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.657254 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.657210    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-service-ca\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.659024 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.658206    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-oauth-serving-cert\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.659024 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.658986    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-trusted-ca-bundle\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.659186 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.659142    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-console-config\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.663392 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.663302    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-serving-cert\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.663664 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.663641    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-oauth-config\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.675773 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.675716    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8p7\" (UniqueName: \"kubernetes.io/projected/78861148-278d-46ef-a9d8-cc188609acc8-kube-api-access-5j8p7\") pod \"console-7c5d7d864c-q6vlv\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") " pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:54.848727 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:54.848497    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:27:55.040663 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.040630    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c5d7d864c-q6vlv"]
Jul 27 19:27:55.046959 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:27:55.046853    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78861148_278d_46ef_a9d8_cc188609acc8.slice/crio-c306469633661c9ee6b520a440beb4a9daeb78d618dd0c426f5319989ccd6a85 WatchSource:0}: Error finding container c306469633661c9ee6b520a440beb4a9daeb78d618dd0c426f5319989ccd6a85: Status 404 returned error can't find the container with id c306469633661c9ee6b520a440beb4a9daeb78d618dd0c426f5319989ccd6a85
Jul 27 19:27:55.519762 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.519416    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298" exitCode=0
Jul 27 19:27:55.519762 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.519511    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"}
Jul 27 19:27:55.523468 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.523444    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5d7d864c-q6vlv" event={"ID":"78861148-278d-46ef-a9d8-cc188609acc8","Type":"ContainerStarted","Data":"383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b"}
Jul 27 19:27:55.523552 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.523482    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5d7d864c-q6vlv" event={"ID":"78861148-278d-46ef-a9d8-cc188609acc8","Type":"ContainerStarted","Data":"c306469633661c9ee6b520a440beb4a9daeb78d618dd0c426f5319989ccd6a85"}
Jul 27 19:27:55.614680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.614626    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c5d7d864c-q6vlv" podStartSLOduration=1.614605864 podStartE2EDuration="1.614605864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:55.613448392 +0000 UTC m=+224.609296384" watchObservedRunningTime="2026-07-27 19:27:55.614605864 +0000 UTC m=+224.610453838"
Jul 27 19:27:55.649735 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.649672    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:55.749892 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:55.749857    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:57.538927 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:57.538889    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"b7b306089edcd5ca1b8c77610bc246e5f1f33c3cfc87b9da813914424daca1c9"}
Jul 27 19:27:57.539356 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:57.538972    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"21aec134b7c962417df6cef018aa5933f44de8c82c8326a50a95f6dfcf7cf1a7"}
Jul 27 19:27:57.539356 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:57.539014    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"86cb95cd37f446ed514903f9f396cc50bb9dc16f92f473ef51c00d266ce08ae1"}
Jul 27 19:27:57.695240 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:57.695202    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:57.802300 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:57.802209    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:27:59.186596 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:27:59.186498    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:28:00.556194 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.556163    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"}
Jul 27 19:28:00.556647 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.556244    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"}
Jul 27 19:28:00.556647 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.556285    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"}
Jul 27 19:28:00.560025 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.559995    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"f1595f3b6f17a4f991bcb4016fd29136f6ec263e63d43ac68bbd89f1b6f622fd"}
Jul 27 19:28:00.560165 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.560082    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"23f58cc669176f9294b4adb29f70efd7c22f7dbace9707f0f54d65d95fdc047f"}
Jul 27 19:28:00.560165 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.560140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" event={"ID":"e12a874c-f838-42cd-98db-2936008d6f86","Type":"ContainerStarted","Data":"2252c64148c5446dd97d94d1eb27ee44d3cc1bb09f8a6ec20dea82bf7506426f"}
Jul 27 19:28:00.561160 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:00.561139    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:28:01.570027 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:01.569990    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"}
Jul 27 19:28:01.570490 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:01.570069    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"}
Jul 27 19:28:01.570490 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:01.570115    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerStarted","Data":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"}
Jul 27 19:28:01.576562 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:01.576516    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd"
Jul 27 19:28:01.613666 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:01.613605    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.572138089 podStartE2EDuration="11.613583527s" totalImagesPullingTime="5.997211693s" totalInitContainerRuntime="44.233745ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:01.61146805 +0000 UTC m=+230.607316027" watchObservedRunningTime="2026-07-27 19:28:01.613583527 +0000 UTC m=+230.609431504"
Jul 27 19:28:01.614282 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:01.614247    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-849d4f79c7-rgckd" podStartSLOduration=8.862750111 podStartE2EDuration="14.614237097s" totalImagesPullingTime="5.751486986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:00.598687847 +0000 UTC m=+229.594535824" watchObservedRunningTime="2026-07-27 19:28:01.614237097 +0000 UTC m=+230.610085074"
Jul 27 19:28:04.848717 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:04.848681    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:28:04.849082 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:04.848768    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:28:04.853363 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:04.853342    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:28:05.589794 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:05.589766    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:28:05.659535 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:05.659490    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777dc76db4-bxs67"]
Jul 27 19:28:05.801710 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:05.801676    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:05.805108 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:05.805089    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:28:05.902016 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:05.901933    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:05.987873 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:05.987847    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:06.791503 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:06.791471    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:06.891517 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:06.891490    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:08.544038 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.543984    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f4f4f8d5b-n2pzh" podUID="18eb1627-e40f-42f7-8cf9-cb58997bd08f" containerName="console" containerID="cri-o://1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961" gracePeriod=15
Jul 27 19:28:08.817546 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.817466    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:28:08.900691 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.900651    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-oauth-serving-cert\") pod \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") "
Jul 27 19:28:08.900691 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.900699    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc5tc\" (UniqueName: \"kubernetes.io/projected/18eb1627-e40f-42f7-8cf9-cb58997bd08f-kube-api-access-fc5tc\") pod \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") "
Jul 27 19:28:08.900924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.900736    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-serving-cert\") pod \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") "
Jul 27 19:28:08.900924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.900760    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-config\") pod \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") "
Jul 27 19:28:08.900924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.900795    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-oauth-config\") pod \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") "
Jul 27 19:28:08.900924 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.900840    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-service-ca\") pod \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\" (UID: \"18eb1627-e40f-42f7-8cf9-cb58997bd08f\") "
Jul 27 19:28:08.901234 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.901085    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "18eb1627-e40f-42f7-8cf9-cb58997bd08f" (UID: "18eb1627-e40f-42f7-8cf9-cb58997bd08f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:08.901234 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.901226    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-config" (OuterVolumeSpecName: "console-config") pod "18eb1627-e40f-42f7-8cf9-cb58997bd08f" (UID: "18eb1627-e40f-42f7-8cf9-cb58997bd08f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:08.901413 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.901258    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-service-ca" (OuterVolumeSpecName: "service-ca") pod "18eb1627-e40f-42f7-8cf9-cb58997bd08f" (UID: "18eb1627-e40f-42f7-8cf9-cb58997bd08f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:08.903221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.903200    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "18eb1627-e40f-42f7-8cf9-cb58997bd08f" (UID: "18eb1627-e40f-42f7-8cf9-cb58997bd08f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:08.903321 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.903218    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "18eb1627-e40f-42f7-8cf9-cb58997bd08f" (UID: "18eb1627-e40f-42f7-8cf9-cb58997bd08f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:08.903321 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:08.903216    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18eb1627-e40f-42f7-8cf9-cb58997bd08f-kube-api-access-fc5tc" (OuterVolumeSpecName: "kube-api-access-fc5tc") pod "18eb1627-e40f-42f7-8cf9-cb58997bd08f" (UID: "18eb1627-e40f-42f7-8cf9-cb58997bd08f"). InnerVolumeSpecName "kube-api-access-fc5tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:28:09.001799 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.001764    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-oauth-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:09.001799 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.001794    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc5tc\" (UniqueName: \"kubernetes.io/projected/18eb1627-e40f-42f7-8cf9-cb58997bd08f-kube-api-access-fc5tc\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:09.001799 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.001804    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:09.002012 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.001812    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:09.002012 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.001820    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18eb1627-e40f-42f7-8cf9-cb58997bd08f-console-oauth-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:09.002012 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.001834    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18eb1627-e40f-42f7-8cf9-cb58997bd08f-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:09.602929 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.602897    2568 generic.go:356] "Generic (PLEG): container finished" podID="18eb1627-e40f-42f7-8cf9-cb58997bd08f" containerID="1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961" exitCode=0
Jul 27 19:28:09.603311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.602946    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4f4f8d5b-n2pzh"
Jul 27 19:28:09.603311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.602948    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4f4f8d5b-n2pzh" event={"ID":"18eb1627-e40f-42f7-8cf9-cb58997bd08f","Type":"ContainerDied","Data":"1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961"}
Jul 27 19:28:09.603311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.602989    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4f4f8d5b-n2pzh" event={"ID":"18eb1627-e40f-42f7-8cf9-cb58997bd08f","Type":"ContainerDied","Data":"31fef50d1d36275b6af92156797f04e954b55f295c3f002fd0629a31bcd7b213"}
Jul 27 19:28:09.603311 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.603007    2568 scope.go:117] "RemoveContainer" containerID="1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961"
Jul 27 19:28:09.611302 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.611283    2568 scope.go:117] "RemoveContainer" containerID="1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961"
Jul 27 19:28:09.611581 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:28:09.611563    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961\": container with ID starting with 1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961 not found: ID does not exist" containerID="1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961"
Jul 27 19:28:09.611639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.611591    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961"} err="failed to get container status \"1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961\": rpc error: code = NotFound desc = could not find container \"1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961\": container with ID starting with 1b559358b70dd75b090614df5cf2bebf18fbc45764b3974c7c9af3fadd644961 not found: ID does not exist"
Jul 27 19:28:09.625656 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.625628    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4f4f8d5b-n2pzh"]
Jul 27 19:28:09.632261 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:09.632241    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f4f4f8d5b-n2pzh"]
Jul 27 19:28:11.571440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:11.571405    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18eb1627-e40f-42f7-8cf9-cb58997bd08f" path="/var/lib/kubelet/pods/18eb1627-e40f-42f7-8cf9-cb58997bd08f/volumes"
Jul 27 19:28:12.689344 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:12.689290    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:12.772030 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:12.771998    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:20.547852 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.547795    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5479c7dc6-kp49j" podUID="deed3143-d011-4e9b-843d-ca0b73042493" containerName="console" containerID="cri-o://bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327" gracePeriod=15
Jul 27 19:28:20.798915 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.798863    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:28:20.910498 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910460    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc76w\" (UniqueName: \"kubernetes.io/projected/deed3143-d011-4e9b-843d-ca0b73042493-kube-api-access-zc76w\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.910694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910503    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-serving-cert\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.910694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910572    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-oauth-config\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.910694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910605    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-trusted-ca-bundle\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.910694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910634    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-console-config\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.910694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910678    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-oauth-serving-cert\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.910938 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.910742    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-service-ca\") pod \"deed3143-d011-4e9b-843d-ca0b73042493\" (UID: \"deed3143-d011-4e9b-843d-ca0b73042493\") "
Jul 27 19:28:20.911097 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.911053    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-console-config" (OuterVolumeSpecName: "console-config") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:20.911097 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.911071    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:20.911097 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.911076    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:20.911463 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.911442    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-service-ca" (OuterVolumeSpecName: "service-ca") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:20.912783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.912753    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:20.912868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.912777    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:20.912868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:20.912808    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deed3143-d011-4e9b-843d-ca0b73042493-kube-api-access-zc76w" (OuterVolumeSpecName: "kube-api-access-zc76w") pod "deed3143-d011-4e9b-843d-ca0b73042493" (UID: "deed3143-d011-4e9b-843d-ca0b73042493"). InnerVolumeSpecName "kube-api-access-zc76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:28:21.011545 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011470    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.011545 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011512    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc76w\" (UniqueName: \"kubernetes.io/projected/deed3143-d011-4e9b-843d-ca0b73042493-kube-api-access-zc76w\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.011545 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011549    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.011771 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011564    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deed3143-d011-4e9b-843d-ca0b73042493-console-oauth-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.011771 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011575    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-trusted-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.011771 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011589    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-console-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.011771 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.011599    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deed3143-d011-4e9b-843d-ca0b73042493-oauth-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:21.649004 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.648973    2568 generic.go:356] "Generic (PLEG): container finished" podID="deed3143-d011-4e9b-843d-ca0b73042493" containerID="bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327" exitCode=0
Jul 27 19:28:21.649394 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.649016    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5479c7dc6-kp49j"
Jul 27 19:28:21.649394 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.649048    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5479c7dc6-kp49j" event={"ID":"deed3143-d011-4e9b-843d-ca0b73042493","Type":"ContainerDied","Data":"bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327"}
Jul 27 19:28:21.649394 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.649092    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5479c7dc6-kp49j" event={"ID":"deed3143-d011-4e9b-843d-ca0b73042493","Type":"ContainerDied","Data":"916e97d777b7b24dd9ecb2e5626f7839eb7a1137ef489568e6a1cdf154c4515f"}
Jul 27 19:28:21.649394 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.649116    2568 scope.go:117] "RemoveContainer" containerID="bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327"
Jul 27 19:28:21.657087 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.657071    2568 scope.go:117] "RemoveContainer" containerID="bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327"
Jul 27 19:28:21.657358 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:28:21.657338    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327\": container with ID starting with bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327 not found: ID does not exist" containerID="bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327"
Jul 27 19:28:21.657410 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.657365    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327"} err="failed to get container status \"bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327\": rpc error: code = NotFound desc = could not find container \"bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327\": container with ID starting with bfe5edd4fed2e61ca1f192d76c482e82964b9bc917212807258afaf063cc2327 not found: ID does not exist"
Jul 27 19:28:21.676042 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.676015    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5479c7dc6-kp49j"]
Jul 27 19:28:21.681395 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:21.681376    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5479c7dc6-kp49j"]
Jul 27 19:28:23.566639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:23.566607    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deed3143-d011-4e9b-843d-ca0b73042493" path="/var/lib/kubelet/pods/deed3143-d011-4e9b-843d-ca0b73042493/volumes"
Jul 27 19:28:30.680080 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:30.680041    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-777dc76db4-bxs67" podUID="fa6271cc-a4dc-4a3d-98bd-3036e6f248db" containerName="console" containerID="cri-o://08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3" gracePeriod=15
Jul 27 19:28:30.955218 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:30.955157    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:28:31.099882 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.099847    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-trusted-ca-bundle\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.099882 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.099884    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-config\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.100122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.099902    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-service-ca\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.100122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.099928    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-oauth-config\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.100122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.099978    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-serving-cert\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.100122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.100016    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmsv4\" (UniqueName: \"kubernetes.io/projected/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-kube-api-access-mmsv4\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.100122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.100041    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-oauth-serving-cert\") pod \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\" (UID: \"fa6271cc-a4dc-4a3d-98bd-3036e6f248db\") "
Jul 27 19:28:31.100436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.100402    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-config" (OuterVolumeSpecName: "console-config") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:31.100515 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.100442    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:31.100643 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.100622    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-service-ca" (OuterVolumeSpecName: "service-ca") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:31.100710 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.100692    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:28:31.102390 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.102359    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-kube-api-access-mmsv4" (OuterVolumeSpecName: "kube-api-access-mmsv4") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "kube-api-access-mmsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:28:31.102744 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.102722    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:31.102744 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.102736    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fa6271cc-a4dc-4a3d-98bd-3036e6f248db" (UID: "fa6271cc-a4dc-4a3d-98bd-3036e6f248db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:28:31.201737 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201706    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-trusted-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.201737 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201731    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.201737 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201740    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.201942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201748    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-oauth-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.201942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201757    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-console-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.201942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201765    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsv4\" (UniqueName: \"kubernetes.io/projected/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-kube-api-access-mmsv4\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.201942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.201774    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa6271cc-a4dc-4a3d-98bd-3036e6f248db-oauth-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:28:31.689507 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.689473    2568 generic.go:356] "Generic (PLEG): container finished" podID="fa6271cc-a4dc-4a3d-98bd-3036e6f248db" containerID="08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3" exitCode=0
Jul 27 19:28:31.689914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.689545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777dc76db4-bxs67" event={"ID":"fa6271cc-a4dc-4a3d-98bd-3036e6f248db","Type":"ContainerDied","Data":"08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3"}
Jul 27 19:28:31.689914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.689549    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777dc76db4-bxs67"
Jul 27 19:28:31.689914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.689588    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777dc76db4-bxs67" event={"ID":"fa6271cc-a4dc-4a3d-98bd-3036e6f248db","Type":"ContainerDied","Data":"bd80867b90bf56d6c2e20fa994d2da27f736ca0c2f96e73f835205218502bcad"}
Jul 27 19:28:31.689914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.689605    2568 scope.go:117] "RemoveContainer" containerID="08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3"
Jul 27 19:28:31.707516 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.707498    2568 scope.go:117] "RemoveContainer" containerID="08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3"
Jul 27 19:28:31.707859 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:28:31.707836    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3\": container with ID starting with 08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3 not found: ID does not exist" containerID="08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3"
Jul 27 19:28:31.707909 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.707869    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3"} err="failed to get container status \"08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3\": rpc error: code = NotFound desc = could not find container \"08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3\": container with ID starting with 08984024a7a52b3e9feab36110b6969cf8fa75db7acdd74e245b25a45588ace3 not found: ID does not exist"
Jul 27 19:28:31.711610 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.711587    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777dc76db4-bxs67"]
Jul 27 19:28:31.716875 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:31.716851    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-777dc76db4-bxs67"]
Jul 27 19:28:33.566806 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:33.566779    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6271cc-a4dc-4a3d-98bd-3036e6f248db" path="/var/lib/kubelet/pods/fa6271cc-a4dc-4a3d-98bd-3036e6f248db/volumes"
Jul 27 19:28:36.257912 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:36.257878    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:28:50.804486 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:50.804449    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:28:50.823566 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:50.823519    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:28:51.781659 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:28:51.781634    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:00.340222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:00.340181    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:00.417334 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:00.417305    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:08.902262 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902226    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:29:08.902687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902500    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="prometheus" containerID="cri-o://d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d" gracePeriod=600
Jul 27 19:29:08.902687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902565    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="thanos-sidecar" containerID="cri-o://48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8" gracePeriod=600
Jul 27 19:29:08.902687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902569    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy" containerID="cri-o://742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9" gracePeriod=600
Jul 27 19:29:08.902687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902589    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="config-reloader" containerID="cri-o://2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d" gracePeriod=600
Jul 27 19:29:08.902687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902625    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8" gracePeriod=600
Jul 27 19:29:08.902687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:08.902642    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-web" containerID="cri-o://5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f" gracePeriod=600
Jul 27 19:29:09.136847 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.136826    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:09.224100 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224019    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-2\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224100 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224051    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-trusted-ca-bundle\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224100 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224079    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-metrics-client-ca\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224110    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224145    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-kubelet-serving-ca-bundle\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224170    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-kube-rbac-proxy\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224194    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-0\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224259    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config-out\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224287    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-thanos-prometheus-http-client-file\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224318    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-serving-certs-ca-bundle\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224345    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbflw\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-kube-api-access-hbflw\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224373    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-tls\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224408    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-grpc-tls\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224433    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224471    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-metrics-client-certs\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224483    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224503    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224499    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-1\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224676    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224696    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-web-config\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224727    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-db\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224753    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224744    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.224774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224779    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224783    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.224818    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-tls-assets\") pod \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\" (UID: \"c01ea71c-c5d0-417c-8b48-744d3f6b75a8\") "
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.225080    2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.225097    2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.225109    2568 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-metrics-client-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.225124    2568 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.225137    2568 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.225571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.225150    2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.226396 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.226366    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:29:09.227295 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227206    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config-out" (OuterVolumeSpecName: "config-out") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:29:09.227405 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227359    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.227405 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227379    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.227565 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227418    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.227735 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227697    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:29:09.227735 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227714    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:29:09.228042 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.227979    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.228792 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.228765    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.229638 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.229543    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.229825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.229805    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config" (OuterVolumeSpecName: "config") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.229910 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.229895    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-kube-api-access-hbflw" (OuterVolumeSpecName: "kube-api-access-hbflw") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "kube-api-access-hbflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:29:09.230000 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.229980    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.239445 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.239426    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-web-config" (OuterVolumeSpecName: "web-config") pod "c01ea71c-c5d0-417c-8b48-744d3f6b75a8" (UID: "c01ea71c-c5d0-417c-8b48-744d3f6b75a8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:29:09.326111 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326075    2568 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-metrics-client-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326111 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326110    2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-web-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326111 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326119    2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-db\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326128    2568 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326136    2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-tls-assets\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326145    2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326153    2568 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-kube-rbac-proxy\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326161    2568 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326169    2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-config-out\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326177    2568 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326186    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbflw\" (UniqueName: \"kubernetes.io/projected/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-kube-api-access-hbflw\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326195    2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326203    2568 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-grpc-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.326281 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.326211    2568 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c01ea71c-c5d0-417c-8b48-744d3f6b75a8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831762    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8" exitCode=0
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831784    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9" exitCode=0
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831791    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f" exitCode=0
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831799    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8" exitCode=0
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831808    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d" exitCode=0
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831815    2568 generic.go:356] "Generic (PLEG): container finished" podID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d" exitCode=0
Jul 27 19:29:09.831839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831836    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831836    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"}
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831920    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"}
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831963    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"}
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.831985    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.832003    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"}
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.832042    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"}
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.832082    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"}
Jul 27 19:29:09.832158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.832121    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c01ea71c-c5d0-417c-8b48-744d3f6b75a8","Type":"ContainerDied","Data":"26af8fce4aaaa4cd661b316d12d6780f0b0bcef5dc4a095ec8f6ba346ed0fdc8"}
Jul 27 19:29:09.839451 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.839433    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.846258 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.846243    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.852748 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.852733    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.859308 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.859259    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:29:09.860097 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.859736    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.866477 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.866459    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.873343 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.873327    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.878222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.878202    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:29:09.880253 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.880241    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.880486 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.880472    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.880560 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.880494    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"} err="failed to get container status \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist"
Jul 27 19:29:09.880560 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.880509    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.880769 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.880740    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.880802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.880775    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"} err="failed to get container status \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist"
Jul 27 19:29:09.880802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.880791    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.881051 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.881037    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.881091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881054    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"} err="failed to get container status \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist"
Jul 27 19:29:09.881091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881066    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.881303 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.881289    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.881353 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881305    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"} err="failed to get container status \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist"
Jul 27 19:29:09.881353 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881316    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.881509 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.881496    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.881571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881511    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"} err="failed to get container status \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist"
Jul 27 19:29:09.881571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881539    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.881750 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.881735    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.881844 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881752    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"} err="failed to get container status \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist"
Jul 27 19:29:09.881844 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881764    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.881958 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:29:09.881943    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.881995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881961    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"} err="failed to get container status \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist"
Jul 27 19:29:09.881995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.881971    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.882178 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882157    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"} err="failed to get container status \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist"
Jul 27 19:29:09.882178 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882177    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.882390 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882370    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"} err="failed to get container status \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist"
Jul 27 19:29:09.882436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882392    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.882649 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882631    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"} err="failed to get container status \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist"
Jul 27 19:29:09.882695 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882651    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.882853 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882838    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"} err="failed to get container status \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist"
Jul 27 19:29:09.882899 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.882853    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.883064 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883046    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"} err="failed to get container status \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist"
Jul 27 19:29:09.883111 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883065    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.883320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883300    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"} err="failed to get container status \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist"
Jul 27 19:29:09.883320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883319    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.883543 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883504    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"} err="failed to get container status \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist"
Jul 27 19:29:09.883543 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883537    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.883753 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883735    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"} err="failed to get container status \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist"
Jul 27 19:29:09.883800 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883753    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.883941 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883925    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"} err="failed to get container status \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist"
Jul 27 19:29:09.883991 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.883941    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.884121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884103    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"} err="failed to get container status \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist"
Jul 27 19:29:09.884193 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884122    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.884348 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884333    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"} err="failed to get container status \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist"
Jul 27 19:29:09.884415 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884349    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.884579 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884562    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"} err="failed to get container status \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist"
Jul 27 19:29:09.884652 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884581    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.884803 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884787    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"} err="failed to get container status \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist"
Jul 27 19:29:09.884860 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884804    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.885003 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884980    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"} err="failed to get container status \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist"
Jul 27 19:29:09.885003 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.884996    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.885189 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885174    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"} err="failed to get container status \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist"
Jul 27 19:29:09.885245 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885190    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.885410 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885387    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"} err="failed to get container status \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist"
Jul 27 19:29:09.885454 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885411    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.885612 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885596    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"} err="failed to get container status \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist"
Jul 27 19:29:09.885658 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885612    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.885784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885768    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"} err="failed to get container status \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist"
Jul 27 19:29:09.885825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885784    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.885985 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885971    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"} err="failed to get container status \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist"
Jul 27 19:29:09.886035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.885985    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.886152 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886136    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"} err="failed to get container status \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist"
Jul 27 19:29:09.886202 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886152    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.886309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886294    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"} err="failed to get container status \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist"
Jul 27 19:29:09.886355 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886311    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.886475 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886461    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"} err="failed to get container status \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist"
Jul 27 19:29:09.886536 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886476    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.886679 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886660    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"} err="failed to get container status \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist"
Jul 27 19:29:09.886752 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886681    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.886889 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886874    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"} err="failed to get container status \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist"
Jul 27 19:29:09.886938 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.886888    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.887053 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887039    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"} err="failed to get container status \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist"
Jul 27 19:29:09.887098 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887054    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.887224 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887200    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"} err="failed to get container status \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist"
Jul 27 19:29:09.887270 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887225    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.887395 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887382    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"} err="failed to get container status \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist"
Jul 27 19:29:09.887440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887396    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.887576 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887561    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"} err="failed to get container status \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist"
Jul 27 19:29:09.887629 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887576    2568 scope.go:117] "RemoveContainer" containerID="6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"
Jul 27 19:29:09.887736 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887722    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8"} err="failed to get container status \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": rpc error: code = NotFound desc = could not find container \"6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8\": container with ID starting with 6df294fed5ae1111e48d58df21a5729fb2fe7afb31b35d5407b31c7023f07ca8 not found: ID does not exist"
Jul 27 19:29:09.887781 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887736    2568 scope.go:117] "RemoveContainer" containerID="742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"
Jul 27 19:29:09.887886 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887873    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9"} err="failed to get container status \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": rpc error: code = NotFound desc = could not find container \"742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9\": container with ID starting with 742fd7d6b0796450a8bef87e0e163ae2a2b3ede764951e644e44fcb3a92eb9d9 not found: ID does not exist"
Jul 27 19:29:09.887933 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.887886    2568 scope.go:117] "RemoveContainer" containerID="5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"
Jul 27 19:29:09.888047 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888033    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f"} err="failed to get container status \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": rpc error: code = NotFound desc = could not find container \"5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f\": container with ID starting with 5fe3065374a5c94235d3016ab6983d1550b2836a688467c518fba5d11b9d555f not found: ID does not exist"
Jul 27 19:29:09.888093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888047    2568 scope.go:117] "RemoveContainer" containerID="48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"
Jul 27 19:29:09.888222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888208    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8"} err="failed to get container status \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": rpc error: code = NotFound desc = could not find container \"48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8\": container with ID starting with 48b9d82520577472ee6d72f0471747c241c321eee7e717c9040c8911f458a3a8 not found: ID does not exist"
Jul 27 19:29:09.888261 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888221    2568 scope.go:117] "RemoveContainer" containerID="2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"
Jul 27 19:29:09.888383 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888369    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d"} err="failed to get container status \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": rpc error: code = NotFound desc = could not find container \"2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d\": container with ID starting with 2cafec7e00824e1142a1717b07b4a9586f3558b5416dfe82e581f109b2a50f6d not found: ID does not exist"
Jul 27 19:29:09.888428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888383    2568 scope.go:117] "RemoveContainer" containerID="d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"
Jul 27 19:29:09.888594 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888573    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d"} err="failed to get container status \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": rpc error: code = NotFound desc = could not find container \"d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d\": container with ID starting with d9b8e281653f6516b0b9e7f1500d5eb7f59aae07837ee53840e90e30382b621d not found: ID does not exist"
Jul 27 19:29:09.888594 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888594    2568 scope.go:117] "RemoveContainer" containerID="de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"
Jul 27 19:29:09.888823 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.888809    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298"} err="failed to get container status \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": rpc error: code = NotFound desc = could not find container \"de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298\": container with ID starting with de04648817848c23fe230d12fc86893c2f7383d039659c882714ae4d7025a298 not found: ID does not exist"
Jul 27 19:29:09.921339 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921314    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921585    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="prometheus"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921596    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="prometheus"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921606    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6271cc-a4dc-4a3d-98bd-3036e6f248db" containerName="console"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921611    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6271cc-a4dc-4a3d-98bd-3036e6f248db" containerName="console"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921621    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-web"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921627    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-web"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921636    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921641    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921649    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="config-reloader"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921653    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="config-reloader"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921659    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="thanos-sidecar"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921664    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="thanos-sidecar"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921671    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-thanos"
Jul 27 19:29:09.921673 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921675    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-thanos"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921681    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18eb1627-e40f-42f7-8cf9-cb58997bd08f" containerName="console"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921686    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="18eb1627-e40f-42f7-8cf9-cb58997bd08f" containerName="console"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921695    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="init-config-reloader"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921699    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="init-config-reloader"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921706    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deed3143-d011-4e9b-843d-ca0b73042493" containerName="console"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921711    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="deed3143-d011-4e9b-843d-ca0b73042493" containerName="console"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921756    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921765    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6271cc-a4dc-4a3d-98bd-3036e6f248db" containerName="console"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921771    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="config-reloader"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921775    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="thanos-sidecar"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921781    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-web"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921785    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="deed3143-d011-4e9b-843d-ca0b73042493" containerName="console"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921791    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="kube-rbac-proxy-thanos"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921797    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" containerName="prometheus"
Jul 27 19:29:09.922079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.921801    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="18eb1627-e40f-42f7-8cf9-cb58997bd08f" containerName="console"
Jul 27 19:29:09.923898 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.923884    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.926995    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.927200    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.927265    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.927537    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.927762    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928272    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hblm8\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928352    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928505    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928595    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928649    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928759    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928806    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.928808    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6a28c1eqm97nm\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.929071    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 19:29:09.929236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.929137    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 19:29:09.932023 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.932003    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 19:29:09.933722 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.933702    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 19:29:09.945188 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:09.945169    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:29:10.030995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.030965    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.030995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.030998    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031184 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031026    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25cc069-254a-49db-966b-cf666ef8932f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031184 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031051    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031184 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031108    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031184 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031160    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031190    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25cc069-254a-49db-966b-cf666ef8932f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzm7z\" (UniqueName: \"kubernetes.io/projected/c25cc069-254a-49db-966b-cf666ef8932f-kube-api-access-lzm7z\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031252    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-config\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031267    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031286    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031303    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031360    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031386    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031419    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031433    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031602 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031450    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031602 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031602 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.031602 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.031518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.132973 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.132892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.132973 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.132936    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.132973 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.132958    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.132973 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.132975    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133114    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133152    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133439 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133335    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133439 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133373    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133439 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133418    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133499    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133543    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25cc069-254a-49db-966b-cf666ef8932f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133593    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133865 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133865 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133669    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.133865 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25cc069-254a-49db-966b-cf666ef8932f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.134005 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133851    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzm7z\" (UniqueName: \"kubernetes.io/projected/c25cc069-254a-49db-966b-cf666ef8932f-kube-api-access-lzm7z\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.134005 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133895    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-config\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.134005 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.133917    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.134829 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.134118    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.134829 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.134388    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.134829 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.134578    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.135712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.135687    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.135916 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.135892    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.136151 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.136120    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.136584 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.136552    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.137675 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.137279    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.137777 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.137729    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c25cc069-254a-49db-966b-cf666ef8932f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.138246 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.138198    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.138332 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.138265    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c25cc069-254a-49db-966b-cf666ef8932f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.138670 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.138626    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.138670 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.138634    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c25cc069-254a-49db-966b-cf666ef8932f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.138885 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.138839    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.138978 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.138917    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.139179 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.139161    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.139338 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.139319    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.139428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.139407    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.140199 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.140184    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c25cc069-254a-49db-966b-cf666ef8932f-config\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.145954 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.145939    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzm7z\" (UniqueName: \"kubernetes.io/projected/c25cc069-254a-49db-966b-cf666ef8932f-kube-api-access-lzm7z\") pod \"prometheus-k8s-0\" (UID: \"c25cc069-254a-49db-966b-cf666ef8932f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.238670 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.238635    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:10.384594 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.384494    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 19:29:10.387822 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:29:10.387795    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25cc069_254a_49db_966b_cf666ef8932f.slice/crio-c3d3a91951c67c2f5c89fbf0553a11300df434a27ae06e10011dc4ee31297ba5 WatchSource:0}: Error finding container c3d3a91951c67c2f5c89fbf0553a11300df434a27ae06e10011dc4ee31297ba5: Status 404 returned error can't find the container with id c3d3a91951c67c2f5c89fbf0553a11300df434a27ae06e10011dc4ee31297ba5
Jul 27 19:29:10.839618 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.839585    2568 generic.go:356] "Generic (PLEG): container finished" podID="c25cc069-254a-49db-966b-cf666ef8932f" containerID="e0ab15c1d03ff0ce86eaa781474be3916690e9cf050afddf58dbe5540aaa869a" exitCode=0
Jul 27 19:29:10.839783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.839672    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerDied","Data":"e0ab15c1d03ff0ce86eaa781474be3916690e9cf050afddf58dbe5540aaa869a"}
Jul 27 19:29:10.839783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:10.839753    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"c3d3a91951c67c2f5c89fbf0553a11300df434a27ae06e10011dc4ee31297ba5"}
Jul 27 19:29:11.372883 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.372811    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:29:11.379694 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.379668    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:29:11.386659 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.386638    2568 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 19:29:11.566542 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.566401    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01ea71c-c5d0-417c-8b48-744d3f6b75a8" path="/var/lib/kubelet/pods/c01ea71c-c5d0-417c-8b48-744d3f6b75a8/volumes"
Jul 27 19:29:11.847168 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.847136    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"f101b2807a43863131c003957d69133250e6e70fee9fd6cd5931067abdff373b"}
Jul 27 19:29:11.847376 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.847211    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"6a251ea9cc2c982275cfc6dded27d6fc94cd37dc99ca11201629a1393b67a167"}
Jul 27 19:29:11.847376 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.847248    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"0e3acee09a23b065b60cada0fba6a468239606434a97e05839a4a51d9e4fb786"}
Jul 27 19:29:11.847376 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.847282    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"48f3c43511fc98ba2ab635eeeeea3814b62a827106ae98da8fca5ac160d6a77f"}
Jul 27 19:29:11.847376 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.847314    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"65e2da1134ff6677b1fc06684aec939f3eb3c1d9ffdbc7037dbc11769fcc61bc"}
Jul 27 19:29:11.847376 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.847349    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c25cc069-254a-49db-966b-cf666ef8932f","Type":"ContainerStarted","Data":"d9179f190f8ad9b91a10a0acebe55689e07d1b1e9bbc19b2b6b9beb72785cc1d"}
Jul 27 19:29:11.886198 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:11.885978    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.867675455 podStartE2EDuration="2.885958524s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.283069ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:11.885357001 +0000 UTC m=+300.881204978" watchObservedRunningTime="2026-07-27 19:29:11.885958524 +0000 UTC m=+300.881806501"
Jul 27 19:29:12.302359 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:12.302300    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:12.376373 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:12.376340    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:15.238964 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:15.238924    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:29:27.693785 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:27.693753    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:27.794196 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:27.794164    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:27.989193 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:27.989107    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:34.914777 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.914742    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56b4754cf4-4qcf6"]
Jul 27 19:29:34.917457 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.917435    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.928619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-trusted-ca-bundle\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.928760 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928650    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-oauth-config\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.928760 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928687    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-service-ca\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.928760 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928718    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-config\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.928760 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928743    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-oauth-serving-cert\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.929693 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928775    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-serving-cert\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.929693 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.928851    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjn9\" (UniqueName: \"kubernetes.io/projected/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-kube-api-access-hwjn9\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:34.930205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.930187    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4754cf4-4qcf6"]
Jul 27 19:29:34.996859 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:34.996828    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:35.029692 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjn9\" (UniqueName: \"kubernetes.io/projected/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-kube-api-access-hwjn9\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.029692 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029692    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-trusted-ca-bundle\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.029934 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029728    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-oauth-config\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.029934 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029760    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-service-ca\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.029934 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029789    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-config\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.029934 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029812    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-oauth-serving-cert\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.029934 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.029871    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-serving-cert\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.030656 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.030615    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-service-ca\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.030813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.030711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-oauth-serving-cert\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.031075 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.031054    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-config\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.031159 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.031138    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-trusted-ca-bundle\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.032296 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.032276    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-oauth-config\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.032560 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.032545    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-serving-cert\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.040583 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.040559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjn9\" (UniqueName: \"kubernetes.io/projected/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-kube-api-access-hwjn9\") pod \"console-56b4754cf4-4qcf6\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") " pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.227731 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.227635    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:35.359263 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.359230    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4754cf4-4qcf6"]
Jul 27 19:29:35.361877 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:29:35.361853    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc35cdf4_07f6_49ac_9a7f_5c078e8b9357.slice/crio-9903629160d8d9531458bef55b410e542b3020c49248f140c18e1b76421c7b15 WatchSource:0}: Error finding container 9903629160d8d9531458bef55b410e542b3020c49248f140c18e1b76421c7b15: Status 404 returned error can't find the container with id 9903629160d8d9531458bef55b410e542b3020c49248f140c18e1b76421c7b15
Jul 27 19:29:35.937092 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.937056    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4754cf4-4qcf6" event={"ID":"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357","Type":"ContainerStarted","Data":"cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00"}
Jul 27 19:29:35.937586 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.937108    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4754cf4-4qcf6" event={"ID":"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357","Type":"ContainerStarted","Data":"9903629160d8d9531458bef55b410e542b3020c49248f140c18e1b76421c7b15"}
Jul 27 19:29:35.959376 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:35.959331    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56b4754cf4-4qcf6" podStartSLOduration=1.959316673 podStartE2EDuration="1.959316673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:29:35.958416467 +0000 UTC m=+324.954264444" watchObservedRunningTime="2026-07-27 19:29:35.959316673 +0000 UTC m=+324.955164650"
Jul 27 19:29:45.228111 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:45.228058    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:45.228803 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:45.228190    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:45.232834 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:45.232811    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:45.976500 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:45.976472    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:29:46.042423 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:46.042392    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c5d7d864c-q6vlv"]
Jul 27 19:29:46.179362 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:46.179332    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:46.279451 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:46.279419    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:46.351629 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:46.351593    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:29:56.219071 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:29:56.219033    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:07.265477 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:07.265436    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:09.317627 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:09.317591    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:10.239303 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:10.239261    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:30:10.255234 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:10.255206    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:30:11.067262 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.067226    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c5d7d864c-q6vlv" podUID="78861148-278d-46ef-a9d8-cc188609acc8" containerName="console" containerID="cri-o://383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b" gracePeriod=15
Jul 27 19:30:11.083048 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.083026    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 19:30:11.303731 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.303703    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:30:11.447514 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447443    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-serving-cert\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447514 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447475    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-service-ca\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447566    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-oauth-config\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447594    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-oauth-serving-cert\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447629    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j8p7\" (UniqueName: \"kubernetes.io/projected/78861148-278d-46ef-a9d8-cc188609acc8-kube-api-access-5j8p7\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447681    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-console-config\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447722    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-trusted-ca-bundle\") pod \"78861148-278d-46ef-a9d8-cc188609acc8\" (UID: \"78861148-278d-46ef-a9d8-cc188609acc8\") "
Jul 27 19:30:11.447984 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447906    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-service-ca" (OuterVolumeSpecName: "service-ca") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:30:11.448022 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.447989    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:30:11.448152 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.448128    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:30:11.448671 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.448646    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-console-config" (OuterVolumeSpecName: "console-config") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:30:11.449735 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.449705    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:30:11.449825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.449736    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:30:11.449825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.449741    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78861148-278d-46ef-a9d8-cc188609acc8-kube-api-access-5j8p7" (OuterVolumeSpecName: "kube-api-access-5j8p7") pod "78861148-278d-46ef-a9d8-cc188609acc8" (UID: "78861148-278d-46ef-a9d8-cc188609acc8"). InnerVolumeSpecName "kube-api-access-5j8p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:30:11.548315 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548287    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:11.548315 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548310    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:11.548315 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548319    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78861148-278d-46ef-a9d8-cc188609acc8-console-oauth-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:11.548577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548327    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-oauth-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:11.548577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548335    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j8p7\" (UniqueName: \"kubernetes.io/projected/78861148-278d-46ef-a9d8-cc188609acc8-kube-api-access-5j8p7\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:11.548577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548343    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-console-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:11.548577 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:11.548350    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78861148-278d-46ef-a9d8-cc188609acc8-trusted-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:30:12.068479 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.068442    2568 generic.go:356] "Generic (PLEG): container finished" podID="78861148-278d-46ef-a9d8-cc188609acc8" containerID="383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b" exitCode=0
Jul 27 19:30:12.068930 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.068488    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5d7d864c-q6vlv"
Jul 27 19:30:12.068930 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.068542    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5d7d864c-q6vlv" event={"ID":"78861148-278d-46ef-a9d8-cc188609acc8","Type":"ContainerDied","Data":"383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b"}
Jul 27 19:30:12.068930 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.068597    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5d7d864c-q6vlv" event={"ID":"78861148-278d-46ef-a9d8-cc188609acc8","Type":"ContainerDied","Data":"c306469633661c9ee6b520a440beb4a9daeb78d618dd0c426f5319989ccd6a85"}
Jul 27 19:30:12.068930 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.068616    2568 scope.go:117] "RemoveContainer" containerID="383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b"
Jul 27 19:30:12.076548 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.076506    2568 scope.go:117] "RemoveContainer" containerID="383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b"
Jul 27 19:30:12.076827 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:30:12.076810    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b\": container with ID starting with 383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b not found: ID does not exist" containerID="383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b"
Jul 27 19:30:12.076876 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.076836    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b"} err="failed to get container status \"383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b\": rpc error: code = NotFound desc = could not find container \"383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b\": container with ID starting with 383c50044edb062c166a102259683bf5041c65c4ab6f9ddfeecbca2fc0ffc80b not found: ID does not exist"
Jul 27 19:30:12.089208 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.089186    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c5d7d864c-q6vlv"]
Jul 27 19:30:12.093605 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:12.093586    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c5d7d864c-q6vlv"]
Jul 27 19:30:13.572832 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:13.572799    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78861148-278d-46ef-a9d8-cc188609acc8" path="/var/lib/kubelet/pods/78861148-278d-46ef-a9d8-cc188609acc8/volumes"
Jul 27 19:30:38.892893 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:38.892861    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:40.248115 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:40.248081    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:42.502148 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:42.502114    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:42.601755 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:42.601729    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:44.220619 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:44.220584    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:44.307011 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:44.306980    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:44.684033 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:44.684002    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:44.785626 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:44.785599    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:46.489153 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:46.489125    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:46.658601 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:46.658568    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:49.488318 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:49.488273    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:49.569984 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:49.569943    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:30:54.895707 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:30:54.895675    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:01.115731 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:01.115700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:01.211567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:01.211510    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:07.744479 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:07.744444    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:07.846276 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:07.846243    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:10.111324 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:10.111291    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:10.544765 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:10.544732    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:10.643901 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:10.643872    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:18.282415 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.282380    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:18.338403 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.338367    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"]
Jul 27 19:31:18.338747 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.338732    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78861148-278d-46ef-a9d8-cc188609acc8" containerName="console"
Jul 27 19:31:18.338868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.338749    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="78861148-278d-46ef-a9d8-cc188609acc8" containerName="console"
Jul 27 19:31:18.338868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.338836    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="78861148-278d-46ef-a9d8-cc188609acc8" containerName="console"
Jul 27 19:31:18.342066 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.342046    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.344811 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.344789    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 19:31:18.346134 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.346114    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 19:31:18.346230 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.346178    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 19:31:18.346230 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.346182    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 19:31:18.346368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.346235    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 19:31:18.346368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.346258    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-thxqs\""
Jul 27 19:31:18.354020 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.354001    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"]
Jul 27 19:31:18.355313 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.355294    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:18.491862 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.491827    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flrc\" (UniqueName: \"kubernetes.io/projected/f715dda3-a4ee-4004-8c48-396b41d0321b-kube-api-access-9flrc\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.491862 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.491865    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f715dda3-a4ee-4004-8c48-396b41d0321b-cert\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.492122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.491907    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f715dda3-a4ee-4004-8c48-396b41d0321b-metrics-cert\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.492122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.491969    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f715dda3-a4ee-4004-8c48-396b41d0321b-manager-config\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.593179 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.593092    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f715dda3-a4ee-4004-8c48-396b41d0321b-metrics-cert\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.593179 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.593136    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f715dda3-a4ee-4004-8c48-396b41d0321b-manager-config\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.593404 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.593268    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flrc\" (UniqueName: \"kubernetes.io/projected/f715dda3-a4ee-4004-8c48-396b41d0321b-kube-api-access-9flrc\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.593404 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.593325    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f715dda3-a4ee-4004-8c48-396b41d0321b-cert\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.593863 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.593829    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f715dda3-a4ee-4004-8c48-396b41d0321b-manager-config\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.595825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.595799    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f715dda3-a4ee-4004-8c48-396b41d0321b-cert\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.595910 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.595799    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f715dda3-a4ee-4004-8c48-396b41d0321b-metrics-cert\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.604355 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.604332    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flrc\" (UniqueName: \"kubernetes.io/projected/f715dda3-a4ee-4004-8c48-396b41d0321b-kube-api-access-9flrc\") pod \"lws-controller-manager-776dc7b77-44wlv\" (UID: \"f715dda3-a4ee-4004-8c48-396b41d0321b\") " pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.650909 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.650875    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:18.782627 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.782594    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"]
Jul 27 19:31:18.785277 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:31:18.785249    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf715dda3_a4ee_4004_8c48_396b41d0321b.slice/crio-b51417290754a34cbaa155bc44cb2570f491364f47b07a04c585c5c8dae096ed WatchSource:0}: Error finding container b51417290754a34cbaa155bc44cb2570f491364f47b07a04c585c5c8dae096ed: Status 404 returned error can't find the container with id b51417290754a34cbaa155bc44cb2570f491364f47b07a04c585c5c8dae096ed
Jul 27 19:31:18.787066 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:18.787048    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:31:19.324353 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:19.324323    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv" event={"ID":"f715dda3-a4ee-4004-8c48-396b41d0321b","Type":"ContainerStarted","Data":"b51417290754a34cbaa155bc44cb2570f491364f47b07a04c585c5c8dae096ed"}
Jul 27 19:31:25.490046 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:25.490006    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:25.591032 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:25.590994    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.159343 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.159311    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.314635 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.314604    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.466224 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.466145    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"]
Jul 27 19:31:26.470642 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.470627    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.473750 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.473728    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 19:31:26.473959 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.473937    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 19:31:26.474073 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.473981    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 19:31:26.474073 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.473998    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 19:31:26.474073 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.474010    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tw2cw\""
Jul 27 19:31:26.486150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.486119    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"]
Jul 27 19:31:26.488034 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.488008    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.553955 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.553916    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a9816b64-77d3-4fb4-8314-982231604886-manifests\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.553955 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.553958    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9816b64-77d3-4fb4-8314-982231604886-tmp\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.554365 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.553985    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9x59\" (UniqueName: \"kubernetes.io/projected/a9816b64-77d3-4fb4-8314-982231604886-kube-api-access-g9x59\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.554365 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.554067    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a9816b64-77d3-4fb4-8314-982231604886-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.554365 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.554105    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9816b64-77d3-4fb4-8314-982231604886-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.622215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.622173    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.654956 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.654919    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a9816b64-77d3-4fb4-8314-982231604886-manifests\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.655139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.654969    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9816b64-77d3-4fb4-8314-982231604886-tmp\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.655139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.655006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9x59\" (UniqueName: \"kubernetes.io/projected/a9816b64-77d3-4fb4-8314-982231604886-kube-api-access-g9x59\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.655139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.655070    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a9816b64-77d3-4fb4-8314-982231604886-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.655139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.655106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9816b64-77d3-4fb4-8314-982231604886-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.655339 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.655314    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a9816b64-77d3-4fb4-8314-982231604886-manifests\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.657424 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.657401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9816b64-77d3-4fb4-8314-982231604886-tmp\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.657679 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.657660    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a9816b64-77d3-4fb4-8314-982231604886-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.657784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.657768    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a9816b64-77d3-4fb4-8314-982231604886-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.667830 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.667801    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9x59\" (UniqueName: \"kubernetes.io/projected/a9816b64-77d3-4fb4-8314-982231604886-kube-api-access-g9x59\") pod \"opendatahub-operator-controller-manager-6c5c9bc998-5kxml\" (UID: \"a9816b64-77d3-4fb4-8314-982231604886\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.720073 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.719993    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.781331 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.781300    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:26.819746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.819720    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.897690 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.897663    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:26.925200 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.925023    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"]
Jul 27 19:31:26.928278 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:31:26.928252    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9816b64_77d3_4fb4_8314_982231604886.slice/crio-f12c6662133e3c355e482a919d1964490104709b1e12bc64ce41ece9475d4be7 WatchSource:0}: Error finding container f12c6662133e3c355e482a919d1964490104709b1e12bc64ce41ece9475d4be7: Status 404 returned error can't find the container with id f12c6662133e3c355e482a919d1964490104709b1e12bc64ce41ece9475d4be7
Jul 27 19:31:26.969274 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:26.969248    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:27.073620 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:27.073549    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:27.171062 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:27.171022    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:27.360259 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:27.360127    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml" event={"ID":"a9816b64-77d3-4fb4-8314-982231604886","Type":"ContainerStarted","Data":"f12c6662133e3c355e482a919d1964490104709b1e12bc64ce41ece9475d4be7"}
Jul 27 19:31:31.380317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:31.380279    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml" event={"ID":"a9816b64-77d3-4fb4-8314-982231604886","Type":"ContainerStarted","Data":"64f2110b79d927a1121fc80b6f803940b2fff326023c84a03cd2508d03e6f661"}
Jul 27 19:31:31.381776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:31.381753    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv" event={"ID":"f715dda3-a4ee-4004-8c48-396b41d0321b","Type":"ContainerStarted","Data":"4604e4b22724ed88ef62ffc4f97971ddb67e5b108a58ea76ac6b00c99b5e19d8"}
Jul 27 19:31:31.381885 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:31.381794    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:31.425594 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:31.425543    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv" podStartSLOduration=1.64768982 podStartE2EDuration="13.425505676s" totalImagesPullingTime="11.777815856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:31:31.425380815 +0000 UTC m=+440.421228792" watchObservedRunningTime="2026-07-27 19:31:31.425505676 +0000 UTC m=+440.421353681"
Jul 27 19:31:32.393130 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:32.393091    2568 generic.go:356] "Generic (PLEG): container finished" podID="a9816b64-77d3-4fb4-8314-982231604886" containerID="64f2110b79d927a1121fc80b6f803940b2fff326023c84a03cd2508d03e6f661" exitCode=0
Jul 27 19:31:32.393632 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:32.393180    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml" event={"ID":"a9816b64-77d3-4fb4-8314-982231604886","Type":"ContainerDied","Data":"64f2110b79d927a1121fc80b6f803940b2fff326023c84a03cd2508d03e6f661"}
Jul 27 19:31:33.399141 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:33.399110    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml" event={"ID":"a9816b64-77d3-4fb4-8314-982231604886","Type":"ContainerStarted","Data":"2b5ca66c01d73a44d44acfc4daf994e83eb163360edfe42fdcf75ddd61bf9cb5"}
Jul 27 19:31:33.399515 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:33.399220    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:33.426735 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:33.426686    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml" podStartSLOduration=2.362517501 podStartE2EDuration="7.426666997s" totalImagesPullingTime="4.036037449s" totalInitContainerRuntime="1.028112047s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:31:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:31:33.424404661 +0000 UTC m=+442.420252643" watchObservedRunningTime="2026-07-27 19:31:33.426666997 +0000 UTC m=+442.422514973"
Jul 27 19:31:37.624428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:37.624357    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:37.721896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:37.721864    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:42.395842 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:42.395807    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-776dc7b77-44wlv"
Jul 27 19:31:44.406509 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:44.406479    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c5c9bc998-5kxml"
Jul 27 19:31:45.917054 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:45.917023    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:46.022511 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.022481    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:46.489736 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.489700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:46.668519 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.668489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:46.767695 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.767671    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"]
Jul 27 19:31:46.778350 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.778326    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.781254 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.781198    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 19:31:46.781545 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.781424    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 19:31:46.781545 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.781483    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 19:31:46.781764 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.781746    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"]
Jul 27 19:31:46.782362 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.782342    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k85j9\""
Jul 27 19:31:46.782469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.782342    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 19:31:46.832610 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.832577    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcrd\" (UniqueName: \"kubernetes.io/projected/7f02614d-835b-4b11-b9a8-22f371c89f23-kube-api-access-qqcrd\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.832787 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.832629    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f02614d-835b-4b11-b9a8-22f371c89f23-tmp\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.832787 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.832646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f02614d-835b-4b11-b9a8-22f371c89f23-tls-certs\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.929954 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.929920    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:46.933109 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.933086    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcrd\" (UniqueName: \"kubernetes.io/projected/7f02614d-835b-4b11-b9a8-22f371c89f23-kube-api-access-qqcrd\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.933179 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.933139    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f02614d-835b-4b11-b9a8-22f371c89f23-tmp\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.933179 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.933158    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f02614d-835b-4b11-b9a8-22f371c89f23-tls-certs\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.935574 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.935552    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f02614d-835b-4b11-b9a8-22f371c89f23-tmp\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.935871 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.935851    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f02614d-835b-4b11-b9a8-22f371c89f23-tls-certs\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:46.950369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:46.950344    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcrd\" (UniqueName: \"kubernetes.io/projected/7f02614d-835b-4b11-b9a8-22f371c89f23-kube-api-access-qqcrd\") pod \"kube-auth-proxy-6bbdd9d5dd-tbrch\" (UID: \"7f02614d-835b-4b11-b9a8-22f371c89f23\") " pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:47.046176 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.046102    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:47.091086 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.091058    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"
Jul 27 19:31:47.149642 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.149614    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:47.235827 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.235703    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch"]
Jul 27 19:31:47.238478 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:31:47.238455    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f02614d_835b_4b11_b9a8_22f371c89f23.slice/crio-a3a26fd68a03723a2f58363d108ab13d2a12ccf98ac2155afe834715fc887094 WatchSource:0}: Error finding container a3a26fd68a03723a2f58363d108ab13d2a12ccf98ac2155afe834715fc887094: Status 404 returned error can't find the container with id a3a26fd68a03723a2f58363d108ab13d2a12ccf98ac2155afe834715fc887094
Jul 27 19:31:47.251567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.251513    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:47.444654 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.444500    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:47.459208 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.459174    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch" event={"ID":"7f02614d-835b-4b11-b9a8-22f371c89f23","Type":"ContainerStarted","Data":"a3a26fd68a03723a2f58363d108ab13d2a12ccf98ac2155afe834715fc887094"}
Jul 27 19:31:47.791703 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.791669    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:47.889466 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.889417    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:47.952369 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:47.952338    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:48.053572 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:48.053469    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:49.468826 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:49.468793    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:50.474121 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:50.474086    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch" event={"ID":"7f02614d-835b-4b11-b9a8-22f371c89f23","Type":"ContainerStarted","Data":"27654503d1ae6c7c25cf9a83d7cc4b942761219f3c6f7e01fc381ae86a53db8b"}
Jul 27 19:31:50.494223 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:50.494164    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bbdd9d5dd-tbrch" podStartSLOduration=1.450350774 podStartE2EDuration="4.494145811s" totalImagesPullingTime="3.043795037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:31:50.493448332 +0000 UTC m=+459.489296308" watchObservedRunningTime="2026-07-27 19:31:50.494145811 +0000 UTC m=+459.489993789"
Jul 27 19:31:53.810295 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:53.810265    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:53.881750 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:53.881718    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:54.548899 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:54.548868    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:54.695501 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:54.695470    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:54.811776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:54.811694    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:57.573682 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:57.573652    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:57.661868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:57.661840    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:57.727801 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:57.727775    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:58.216658 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:58.216626    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:58.419445 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:58.419406    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:58.867957 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:58.867925    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:58.968308 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:58.968268    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:31:59.031076 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:31:59.031048    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:14.310736 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:14.310708    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:29.918136 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:29.918100    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:29.975413 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:29.975383    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:47.447066 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:47.447037    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:47.547445 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:47.547407    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:47.612806 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:47.612776    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:48.390205 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:48.390170    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:32:48.565169 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:32:48.565143    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:09.367466 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:09.367432    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:09.430109 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:09.430072    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:11.213598 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:11.213566    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:11.313019 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:11.312987    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:16.428649 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:16.428618    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:19.827378 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:19.827347    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:20.261820 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:20.261743    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:21.623684 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:21.623650    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:22.038993 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:22.038966    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:23.059914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:23.059883    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:23.162723 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:23.162686    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:24.627592 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:24.627563    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:25.673981 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:25.673946    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:27.069806 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:27.069776    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:28.264840 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.264806    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:28.779542 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.779496    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"]
Jul 27 19:33:28.781715 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.781697    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:28.786019 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.786000    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8z62s\""
Jul 27 19:33:28.787259 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.787236    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 19:33:28.787368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.787276    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 19:33:28.787368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.787281    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 19:33:28.797734 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.797711    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"]
Jul 27 19:33:28.909937 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:28.909904    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jj9\" (UniqueName: \"kubernetes.io/projected/91fcffbe-6d2a-4e46-8aed-f47a152f550e-kube-api-access-z8jj9\") pod \"dns-operator-controller-manager-864dd78c4c-8qbnf\" (UID: \"91fcffbe-6d2a-4e46-8aed-f47a152f550e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:29.011200 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:29.011163    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jj9\" (UniqueName: \"kubernetes.io/projected/91fcffbe-6d2a-4e46-8aed-f47a152f550e-kube-api-access-z8jj9\") pod \"dns-operator-controller-manager-864dd78c4c-8qbnf\" (UID: \"91fcffbe-6d2a-4e46-8aed-f47a152f550e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:29.026125 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:29.026098    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jj9\" (UniqueName: \"kubernetes.io/projected/91fcffbe-6d2a-4e46-8aed-f47a152f550e-kube-api-access-z8jj9\") pod \"dns-operator-controller-manager-864dd78c4c-8qbnf\" (UID: \"91fcffbe-6d2a-4e46-8aed-f47a152f550e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:29.091897 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:29.091811    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:29.233681 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:29.233651    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"]
Jul 27 19:33:29.236868 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:33:29.236840    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fcffbe_6d2a_4e46_8aed_f47a152f550e.slice/crio-68a7b98d53981311d02e4d322b313d42682373d359f7c9d07b6d56da3540dede WatchSource:0}: Error finding container 68a7b98d53981311d02e4d322b313d42682373d359f7c9d07b6d56da3540dede: Status 404 returned error can't find the container with id 68a7b98d53981311d02e4d322b313d42682373d359f7c9d07b6d56da3540dede
Jul 27 19:33:29.888953 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:29.888920    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf" event={"ID":"91fcffbe-6d2a-4e46-8aed-f47a152f550e","Type":"ContainerStarted","Data":"68a7b98d53981311d02e4d322b313d42682373d359f7c9d07b6d56da3540dede"}
Jul 27 19:33:30.851552 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:30.851511    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:30.949803 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:30.949760    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:31.899135 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:31.899043    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf" event={"ID":"91fcffbe-6d2a-4e46-8aed-f47a152f550e","Type":"ContainerStarted","Data":"e6e0021e6c39f7b53529a144a92d42aeb8a66dd983d671e603941a61eb6fff66"}
Jul 27 19:33:31.899135 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:31.899110    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:31.920433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:31.920388    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf" podStartSLOduration=1.576662714 podStartE2EDuration="3.920371879s" totalImagesPullingTime="2.343709165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:33:31.918988643 +0000 UTC m=+560.914836620" watchObservedRunningTime="2026-07-27 19:33:31.920371879 +0000 UTC m=+560.916219856"
Jul 27 19:33:32.299653 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.299618    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"]
Jul 27 19:33:32.301925 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.301903    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.304845 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.304827    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 27 19:33:32.304946 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.304919    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6r6qb\""
Jul 27 19:33:32.304995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.304978    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 27 19:33:32.313349 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.313329    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"]
Jul 27 19:33:32.316987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.316968    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:32.443050 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.443012    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxlp\" (UniqueName: \"kubernetes.io/projected/1fe84938-3a52-4844-8351-46474577827f-kube-api-access-dzxlp\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.443215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.443059    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fe84938-3a52-4844-8351-46474577827f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.443215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.443153    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe84938-3a52-4844-8351-46474577827f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.459304 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.459270    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:32.544035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.544001    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe84938-3a52-4844-8351-46474577827f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.544240 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.544058    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxlp\" (UniqueName: \"kubernetes.io/projected/1fe84938-3a52-4844-8351-46474577827f-kube-api-access-dzxlp\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.544240 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.544089    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fe84938-3a52-4844-8351-46474577827f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.544827 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.544798    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1fe84938-3a52-4844-8351-46474577827f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.546690 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.546672    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fe84938-3a52-4844-8351-46474577827f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.558315 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.558267    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxlp\" (UniqueName: \"kubernetes.io/projected/1fe84938-3a52-4844-8351-46474577827f-kube-api-access-dzxlp\") pod \"kuadrant-console-plugin-66b5774985-7x6j2\" (UID: \"1fe84938-3a52-4844-8351-46474577827f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.611269 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.611240    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"
Jul 27 19:33:32.751051 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.751024    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2"]
Jul 27 19:33:32.753336 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:33:32.753312    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe84938_3a52_4844_8351_46474577827f.slice/crio-bbd6b7cd2a81f0fe3c33c5f8c5225f58c7fd03285cdee46ed2aa95df8385ac3d WatchSource:0}: Error finding container bbd6b7cd2a81f0fe3c33c5f8c5225f58c7fd03285cdee46ed2aa95df8385ac3d: Status 404 returned error can't find the container with id bbd6b7cd2a81f0fe3c33c5f8c5225f58c7fd03285cdee46ed2aa95df8385ac3d
Jul 27 19:33:32.910571 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:32.910457    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2" event={"ID":"1fe84938-3a52-4844-8351-46474577827f","Type":"ContainerStarted","Data":"bbd6b7cd2a81f0fe3c33c5f8c5225f58c7fd03285cdee46ed2aa95df8385ac3d"}
Jul 27 19:33:34.783790 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:34.783757    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:40.867224 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:40.867187    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:40.971944 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:40.971910    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:42.913182 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:42.913150    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8qbnf"
Jul 27 19:33:43.887541 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:43.887493    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:43.988984 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:43.988947    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:46.310165 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:46.310134    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:46.415753 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:46.415717    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:53.703616 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:53.703553    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:53.780302 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:53.780256    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:33:55.025760 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:55.025727    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2" event={"ID":"1fe84938-3a52-4844-8351-46474577827f","Type":"ContainerStarted","Data":"2025d85c73a5c62ba9533f6bba6eb7cfe44f47740e58753864c3c0db8b34b985"}
Jul 27 19:33:55.044768 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:33:55.044721    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7x6j2" podStartSLOduration=1.406077452 podStartE2EDuration="23.044705241s" totalImagesPullingTime="21.638627789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:33:55.043488099 +0000 UTC m=+584.039336077" watchObservedRunningTime="2026-07-27 19:33:55.044705241 +0000 UTC m=+584.040553218"
Jul 27 19:34:02.348972 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:02.348938    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:02.449534 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:02.449479    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:02.682721 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:02.682635    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:02.782292 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:02.782261    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:05.683127 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:05.683095    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:05.787033 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:05.787003    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:08.118737 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:08.118705    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:08.177131 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:08.177103    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:10.147448 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:10.147417    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:11.403682 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:11.403653    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:34:11.412654 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:11.412634    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:34:12.029336 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:12.029288    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:12.130722 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:12.130691    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:13.825227 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.825192    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:13.836105 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.836081    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:13.838353 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.838340    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:13.844064 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.844043    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 19:34:13.850873 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.850852    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:13.912455 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.912426    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/31db070c-11be-464e-bebd-baa860192cc4-kube-api-access-frtlh\") pod \"limitador-limitador-64c94fd64f-klm9d\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:13.912643 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.912465    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31db070c-11be-464e-bebd-baa860192cc4-config-file\") pod \"limitador-limitador-64c94fd64f-klm9d\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:13.930980 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.930950    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:13.939107 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:13.939081    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:14.009785 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.009750    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:14.013901 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.013883    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/31db070c-11be-464e-bebd-baa860192cc4-kube-api-access-frtlh\") pod \"limitador-limitador-64c94fd64f-klm9d\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:14.013983 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.013915    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31db070c-11be-464e-bebd-baa860192cc4-config-file\") pod \"limitador-limitador-64c94fd64f-klm9d\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:14.014544 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.014503    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31db070c-11be-464e-bebd-baa860192cc4-config-file\") pod \"limitador-limitador-64c94fd64f-klm9d\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:14.023740 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.023718    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/31db070c-11be-464e-bebd-baa860192cc4-kube-api-access-frtlh\") pod \"limitador-limitador-64c94fd64f-klm9d\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:14.147725 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.147647    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:14.280079 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.280050    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:14.283419 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:34:14.283395    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31db070c_11be_464e_bebd_baa860192cc4.slice/crio-079ecd47629f092b74aeacddffdc541b8e56d44fe9b2b0a2eb7bb031fae14bc6 WatchSource:0}: Error finding container 079ecd47629f092b74aeacddffdc541b8e56d44fe9b2b0a2eb7bb031fae14bc6: Status 404 returned error can't find the container with id 079ecd47629f092b74aeacddffdc541b8e56d44fe9b2b0a2eb7bb031fae14bc6
Jul 27 19:34:14.698807 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.698775    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:14.802705 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:14.802675    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:15.112406 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:15.112368    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d" event={"ID":"31db070c-11be-464e-bebd-baa860192cc4","Type":"ContainerStarted","Data":"079ecd47629f092b74aeacddffdc541b8e56d44fe9b2b0a2eb7bb031fae14bc6"}
Jul 27 19:34:18.130087 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:18.130043    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d" event={"ID":"31db070c-11be-464e-bebd-baa860192cc4","Type":"ContainerStarted","Data":"513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd"}
Jul 27 19:34:18.130488 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:18.130186    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:18.151718 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:18.151667    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d" podStartSLOduration=2.3650767950000002 podStartE2EDuration="5.151649193s" totalImagesPullingTime="2.786572398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:34:18.149087701 +0000 UTC m=+607.144935701" watchObservedRunningTime="2026-07-27 19:34:18.151649193 +0000 UTC m=+607.147497171"
Jul 27 19:34:28.489177 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:28.489139    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:28.489643 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:28.489319    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d" podUID="31db070c-11be-464e-bebd-baa860192cc4" containerName="limitador" containerID="cri-o://513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd" gracePeriod=30
Jul 27 19:34:28.489961 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:28.489938    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:29.037608 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.037583    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:29.153278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.153186    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/31db070c-11be-464e-bebd-baa860192cc4-kube-api-access-frtlh\") pod \"31db070c-11be-464e-bebd-baa860192cc4\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") "
Jul 27 19:34:29.153278 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.153254    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31db070c-11be-464e-bebd-baa860192cc4-config-file\") pod \"31db070c-11be-464e-bebd-baa860192cc4\" (UID: \"31db070c-11be-464e-bebd-baa860192cc4\") "
Jul 27 19:34:29.153687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.153663    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31db070c-11be-464e-bebd-baa860192cc4-config-file" (OuterVolumeSpecName: "config-file") pod "31db070c-11be-464e-bebd-baa860192cc4" (UID: "31db070c-11be-464e-bebd-baa860192cc4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:34:29.155631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.155605    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31db070c-11be-464e-bebd-baa860192cc4-kube-api-access-frtlh" (OuterVolumeSpecName: "kube-api-access-frtlh") pod "31db070c-11be-464e-bebd-baa860192cc4" (UID: "31db070c-11be-464e-bebd-baa860192cc4"). InnerVolumeSpecName "kube-api-access-frtlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:34:29.176381 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.176346    2568 generic.go:356] "Generic (PLEG): container finished" podID="31db070c-11be-464e-bebd-baa860192cc4" containerID="513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd" exitCode=0
Jul 27 19:34:29.176512 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.176413    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d"
Jul 27 19:34:29.176512 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.176437    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d" event={"ID":"31db070c-11be-464e-bebd-baa860192cc4","Type":"ContainerDied","Data":"513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd"}
Jul 27 19:34:29.176604 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.176514    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klm9d" event={"ID":"31db070c-11be-464e-bebd-baa860192cc4","Type":"ContainerDied","Data":"079ecd47629f092b74aeacddffdc541b8e56d44fe9b2b0a2eb7bb031fae14bc6"}
Jul 27 19:34:29.176604 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.176555    2568 scope.go:117] "RemoveContainer" containerID="513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd"
Jul 27 19:34:29.185402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.185387    2568 scope.go:117] "RemoveContainer" containerID="513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd"
Jul 27 19:34:29.185676 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:34:29.185657    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd\": container with ID starting with 513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd not found: ID does not exist" containerID="513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd"
Jul 27 19:34:29.185750 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.185688    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd"} err="failed to get container status \"513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd\": rpc error: code = NotFound desc = could not find container \"513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd\": container with ID starting with 513bcc465c793422eee0ed9c52d5aa0c9cbba99d278d6840bc41c715695f1bfd not found: ID does not exist"
Jul 27 19:34:29.199503 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.199484    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:29.205303 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.205284    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klm9d"]
Jul 27 19:34:29.254746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.254724    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtlh\" (UniqueName: \"kubernetes.io/projected/31db070c-11be-464e-bebd-baa860192cc4-kube-api-access-frtlh\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:34:29.254746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.254746    2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31db070c-11be-464e-bebd-baa860192cc4-config-file\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:34:29.567364 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.567326    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31db070c-11be-464e-bebd-baa860192cc4" path="/var/lib/kubelet/pods/31db070c-11be-464e-bebd-baa860192cc4/volumes"
Jul 27 19:34:29.892935 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.892858    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:29.999172 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:29.999142    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:30.302516 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.302483    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"]
Jul 27 19:34:30.302866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.302851    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31db070c-11be-464e-bebd-baa860192cc4" containerName="limitador"
Jul 27 19:34:30.302923 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.302868    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="31db070c-11be-464e-bebd-baa860192cc4" containerName="limitador"
Jul 27 19:34:30.302955 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.302931    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="31db070c-11be-464e-bebd-baa860192cc4" containerName="limitador"
Jul 27 19:34:30.305538 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.305511    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.308358 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.308336    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 19:34:30.308491 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.308377    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 19:34:30.309668 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.309646    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zp2qm\""
Jul 27 19:34:30.309668 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.309662    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 27 19:34:30.314500 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.314478    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"]
Jul 27 19:34:30.323511 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.323491    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:30.464896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.464860    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rhw\" (UniqueName: \"kubernetes.io/projected/d3723976-c34f-41ba-9b19-83d330df2ce7-kube-api-access-48rhw\") pod \"postgres-69fb4594cf-vcjnk\" (UID: \"d3723976-c34f-41ba-9b19-83d330df2ce7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.464896 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.464899    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d3723976-c34f-41ba-9b19-83d330df2ce7-data\") pod \"postgres-69fb4594cf-vcjnk\" (UID: \"d3723976-c34f-41ba-9b19-83d330df2ce7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.565936 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.565855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rhw\" (UniqueName: \"kubernetes.io/projected/d3723976-c34f-41ba-9b19-83d330df2ce7-kube-api-access-48rhw\") pod \"postgres-69fb4594cf-vcjnk\" (UID: \"d3723976-c34f-41ba-9b19-83d330df2ce7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.565936 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.565892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d3723976-c34f-41ba-9b19-83d330df2ce7-data\") pod \"postgres-69fb4594cf-vcjnk\" (UID: \"d3723976-c34f-41ba-9b19-83d330df2ce7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.566239 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.566225    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d3723976-c34f-41ba-9b19-83d330df2ce7-data\") pod \"postgres-69fb4594cf-vcjnk\" (UID: \"d3723976-c34f-41ba-9b19-83d330df2ce7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.581341 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.581322    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rhw\" (UniqueName: \"kubernetes.io/projected/d3723976-c34f-41ba-9b19-83d330df2ce7-kube-api-access-48rhw\") pod \"postgres-69fb4594cf-vcjnk\" (UID: \"d3723976-c34f-41ba-9b19-83d330df2ce7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.616225 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.616196    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:30.746478 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:30.746452    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"]
Jul 27 19:34:30.748985 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:34:30.748959    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3723976_c34f_41ba_9b19_83d330df2ce7.slice/crio-24bff698ccf1a37a5a735984ba725a24625b6f1340eca8fcf9a39af4e41380d9 WatchSource:0}: Error finding container 24bff698ccf1a37a5a735984ba725a24625b6f1340eca8fcf9a39af4e41380d9: Status 404 returned error can't find the container with id 24bff698ccf1a37a5a735984ba725a24625b6f1340eca8fcf9a39af4e41380d9
Jul 27 19:34:31.187204 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:31.187173    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk" event={"ID":"d3723976-c34f-41ba-9b19-83d330df2ce7","Type":"ContainerStarted","Data":"24bff698ccf1a37a5a735984ba725a24625b6f1340eca8fcf9a39af4e41380d9"}
Jul 27 19:34:36.213217 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:36.213183    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk" event={"ID":"d3723976-c34f-41ba-9b19-83d330df2ce7","Type":"ContainerStarted","Data":"5a05908f67b024f858f56fce8a2fd4636033dcfcaad51f98f71cb0c0054b9e99"}
Jul 27 19:34:36.213696 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:36.213299    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:36.234584 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:36.234544    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk" podStartSLOduration=0.943148612 podStartE2EDuration="6.234510707s" totalImagesPullingTime="5.291362095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:34:36.234037515 +0000 UTC m=+625.229885495" watchObservedRunningTime="2026-07-27 19:34:36.234510707 +0000 UTC m=+625.230358684"
Jul 27 19:34:42.246453 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:42.246426    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vcjnk"
Jul 27 19:34:42.555472 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:42.555443    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:42.683690 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:42.683657    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:49.646122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:49.646089    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:49.747296 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:49.747261    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:49.808307 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:49.808274    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:49.925812 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:49.925730    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:52.942231 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:52.942192    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:34:53.019100 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:34:53.019068    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:11.993872 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:11.993835    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:25.012326 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.012289    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:25.096657 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.096627    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:25.490959 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.490932    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:25.813198 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.813164    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"]
Jul 27 19:35:25.831108 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.831078    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"]
Jul 27 19:35:25.831279 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.831206    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.834398 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.834378    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pfjtp\""
Jul 27 19:35:25.834516 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.834376    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 19:35:25.851828 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.851806    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd29q\" (UniqueName: \"kubernetes.io/projected/e96386aa-a353-4998-b7a1-15b1b8c38496-kube-api-access-sd29q\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.851922 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.851846    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96386aa-a353-4998-b7a1-15b1b8c38496-maas-api-tls\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.851922 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.851876    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96386aa-a353-4998-b7a1-15b1b8c38496-openshift-service-ca\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.931388 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.931348    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:25.952561 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.952517    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd29q\" (UniqueName: \"kubernetes.io/projected/e96386aa-a353-4998-b7a1-15b1b8c38496-kube-api-access-sd29q\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.952680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.952588    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96386aa-a353-4998-b7a1-15b1b8c38496-maas-api-tls\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.952680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.952632    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96386aa-a353-4998-b7a1-15b1b8c38496-openshift-service-ca\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.953247 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.953224    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96386aa-a353-4998-b7a1-15b1b8c38496-openshift-service-ca\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.955089 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.955063    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e96386aa-a353-4998-b7a1-15b1b8c38496-maas-api-tls\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:25.963046 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:25.963027    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd29q\" (UniqueName: \"kubernetes.io/projected/e96386aa-a353-4998-b7a1-15b1b8c38496-kube-api-access-sd29q\") pod \"maas-api-84cf8569bc-4wtjd\" (UID: \"e96386aa-a353-4998-b7a1-15b1b8c38496\") " pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:26.143866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.143781    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:26.202846 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.202788    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:26.290380 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.290357    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"]
Jul 27 19:35:26.292600 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:35:26.292572    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96386aa_a353_4998_b7a1_15b1b8c38496.slice/crio-223fa23b38118761a61db2676e7f35f564da2306a823515feee12afc1c31e05a WatchSource:0}: Error finding container 223fa23b38118761a61db2676e7f35f564da2306a823515feee12afc1c31e05a: Status 404 returned error can't find the container with id 223fa23b38118761a61db2676e7f35f564da2306a823515feee12afc1c31e05a
Jul 27 19:35:26.296513 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.296492    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:26.426472 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.426385    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd" event={"ID":"e96386aa-a353-4998-b7a1-15b1b8c38496","Type":"ContainerStarted","Data":"223fa23b38118761a61db2676e7f35f564da2306a823515feee12afc1c31e05a"}
Jul 27 19:35:26.505567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.505514    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dsffp"]
Jul 27 19:35:26.508591 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.508574    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.511592 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.511572    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 19:35:26.511760 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.511745    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4rl86\""
Jul 27 19:35:26.519176 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.519155    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dsffp"]
Jul 27 19:35:26.557062 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.557021    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b8f564-3897-4a48-aaa0-4faa317ca28d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dsffp\" (UID: \"a5b8f564-3897-4a48-aaa0-4faa317ca28d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.557209 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.557086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxbs\" (UniqueName: \"kubernetes.io/projected/a5b8f564-3897-4a48-aaa0-4faa317ca28d-kube-api-access-cmxbs\") pod \"payload-processing-5cc6cd444b-dsffp\" (UID: \"a5b8f564-3897-4a48-aaa0-4faa317ca28d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.658244 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.658218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b8f564-3897-4a48-aaa0-4faa317ca28d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dsffp\" (UID: \"a5b8f564-3897-4a48-aaa0-4faa317ca28d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.658393 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.658261    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxbs\" (UniqueName: \"kubernetes.io/projected/a5b8f564-3897-4a48-aaa0-4faa317ca28d-kube-api-access-cmxbs\") pod \"payload-processing-5cc6cd444b-dsffp\" (UID: \"a5b8f564-3897-4a48-aaa0-4faa317ca28d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.658923 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.658904    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b8f564-3897-4a48-aaa0-4faa317ca28d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-dsffp\" (UID: \"a5b8f564-3897-4a48-aaa0-4faa317ca28d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.669454 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.669426    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxbs\" (UniqueName: \"kubernetes.io/projected/a5b8f564-3897-4a48-aaa0-4faa317ca28d-kube-api-access-cmxbs\") pod \"payload-processing-5cc6cd444b-dsffp\" (UID: \"a5b8f564-3897-4a48-aaa0-4faa317ca28d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.819922 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.819891    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:26.977999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:26.977974    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-dsffp"]
Jul 27 19:35:26.980248 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:35:26.980215    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b8f564_3897_4a48_aaa0_4faa317ca28d.slice/crio-3acff73c98a47005eb66c1ba181b2e908c1d73176027555c986acac614cb4234 WatchSource:0}: Error finding container 3acff73c98a47005eb66c1ba181b2e908c1d73176027555c986acac614cb4234: Status 404 returned error can't find the container with id 3acff73c98a47005eb66c1ba181b2e908c1d73176027555c986acac614cb4234
Jul 27 19:35:27.435088 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:27.435059    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp" event={"ID":"a5b8f564-3897-4a48-aaa0-4faa317ca28d","Type":"ContainerStarted","Data":"3acff73c98a47005eb66c1ba181b2e908c1d73176027555c986acac614cb4234"}
Jul 27 19:35:29.451070 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:29.451027    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd" event={"ID":"e96386aa-a353-4998-b7a1-15b1b8c38496","Type":"ContainerStarted","Data":"9febe1b79cee77677b916f8d028f428ede1f26bbd74aef2537cf547394b56cde"}
Jul 27 19:35:29.451491 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:29.451195    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:29.473410 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:29.473353    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd" podStartSLOduration=2.391347502 podStartE2EDuration="4.47333761s" totalImagesPullingTime="2.081990108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:35:29.471575996 +0000 UTC m=+678.467423974" watchObservedRunningTime="2026-07-27 19:35:29.47333761 +0000 UTC m=+678.469185601"
Jul 27 19:35:30.458265 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:30.458234    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp" event={"ID":"a5b8f564-3897-4a48-aaa0-4faa317ca28d","Type":"ContainerStarted","Data":"6fc6866bc6ab72b81cbfbcc938f1c40e8d6b9293acd8846235b576116dbddb5f"}
Jul 27 19:35:30.458848 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:30.458484    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:30.481928 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:30.481875    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp" podStartSLOduration=1.464109397 podStartE2EDuration="4.481855383s" totalImagesPullingTime="3.017745986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:35:30.478795291 +0000 UTC m=+679.474643267" watchObservedRunningTime="2026-07-27 19:35:30.481855383 +0000 UTC m=+679.477703362"
Jul 27 19:35:35.463786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:35.463757    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84cf8569bc-4wtjd"
Jul 27 19:35:36.446691 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:36.446655    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:36.545251 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:36.545208    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:41.465915 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:41.465888    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-dsffp"
Jul 27 19:35:43.172909 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:43.172873    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:53.749622 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.749590    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"]
Jul 27 19:35:53.756064 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.756047    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:53.759109 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.759092    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 19:35:53.759207 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.759106    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 19:35:53.760611 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.760583    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 27 19:35:53.760730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.760611    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l7pwx\""
Jul 27 19:35:53.763134 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.763114    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"]
Jul 27 19:35:53.784883 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.784862    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:35:53.903587 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.903556    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:53.903587 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.903587    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:53.903780 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.903623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:53.903780 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.903702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:53.903780 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.903730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4ms\" (UniqueName: \"kubernetes.io/projected/3ececa3a-ca84-4e35-aa82-6386ec88a69b-kube-api-access-sf4ms\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:53.903780 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:53.903774    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ececa3a-ca84-4e35-aa82-6386ec88a69b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.004703 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.004625    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.004703 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.004664    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.004703 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.004683    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4ms\" (UniqueName: \"kubernetes.io/projected/3ececa3a-ca84-4e35-aa82-6386ec88a69b-kube-api-access-sf4ms\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.004952 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.004720    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ececa3a-ca84-4e35-aa82-6386ec88a69b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.004952 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.004757    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.004952 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.004773    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.005122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.005081    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.005173 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.005154    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.005297 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.005275    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.007041 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.007023    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ececa3a-ca84-4e35-aa82-6386ec88a69b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.007362 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.007347    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ececa3a-ca84-4e35-aa82-6386ec88a69b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.013989 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.013971    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4ms\" (UniqueName: \"kubernetes.io/projected/3ececa3a-ca84-4e35-aa82-6386ec88a69b-kube-api-access-sf4ms\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xggcf\" (UID: \"3ececa3a-ca84-4e35-aa82-6386ec88a69b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.066579 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.066555    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:35:54.201368 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.201345    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"]
Jul 27 19:35:54.204159 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:35:54.204111    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ececa3a_ca84_4e35_aa82_6386ec88a69b.slice/crio-c872bbc7ae30a4e9c9b1a63c70850de23f75154d56547d43779f8ff276e66737 WatchSource:0}: Error finding container c872bbc7ae30a4e9c9b1a63c70850de23f75154d56547d43779f8ff276e66737: Status 404 returned error can't find the container with id c872bbc7ae30a4e9c9b1a63c70850de23f75154d56547d43779f8ff276e66737
Jul 27 19:35:54.569503 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:54.569422    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf" event={"ID":"3ececa3a-ca84-4e35-aa82-6386ec88a69b","Type":"ContainerStarted","Data":"c872bbc7ae30a4e9c9b1a63c70850de23f75154d56547d43779f8ff276e66737"}
Jul 27 19:35:55.574605 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:55.574561    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf" event={"ID":"3ececa3a-ca84-4e35-aa82-6386ec88a69b","Type":"ContainerStarted","Data":"58fb34297ee5e42f7b20bca5394c29d07faf7a67a409173960cb9a06f19adb0f"}
Jul 27 19:35:55.596352 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:35:55.596271    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf" podStartSLOduration=1.587490096 podStartE2EDuration="2.596255224s" totalImagesPullingTime="1.008765128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:35:55.59521742 +0000 UTC m=+704.591065397" watchObservedRunningTime="2026-07-27 19:35:55.596255224 +0000 UTC m=+704.592103202"
Jul 27 19:36:04.067347 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:04.067315    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:36:04.067812 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:04.067395    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:36:04.079451 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:04.079432    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:36:04.622136 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:04.622111    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xggcf"
Jul 27 19:36:06.787984 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:06.787950    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:36:15.135238 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:15.135207    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:36:17.584263 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:17.584221    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:36:28.776140 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:28.776080    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:36:50.696875 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:36:50.696821    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:37:59.937969 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:37:59.937884    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:38:00.032786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:38:00.032750    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:38:01.054072 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:38:01.054040    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:38:01.143093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:38:01.143059    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:39:01.357759 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:39:01.357722    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:39:11.431758 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:39:11.431732    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:39:11.438034 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:39:11.438008    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:39:42.924359 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:39:42.924324    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:39:43.026671 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:39:43.026637    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:39:47.153255 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:39:47.153227    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:05.148563 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:05.148506    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:06.176397 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:06.176357    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:41.668979 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:41.668888    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:43.041100 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:43.041069    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:48.315562 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:48.315511    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:48.374795 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:48.374769    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:48.457095 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:48.457067    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:48.558646 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:48.558616    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:48.692334 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:48.692251    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:55.101541 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:55.101497    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:40:55.168232 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:40:55.168202    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:41:43.707744 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:41:43.707705    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:41:43.811320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:41:43.811291    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:42:36.818839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:42:36.818807    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:44:11.460728 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:44:11.460701    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:44:11.463459 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:44:11.463388    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:46:05.739699 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:46:05.739670    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:48:40.259914 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.259883    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"]
Jul 27 19:48:40.263221 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.263204    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.266063 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.266043    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5a1da6e4-kserve-self-signed-certs\""
Jul 27 19:48:40.284261 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.284225    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"]
Jul 27 19:48:40.331665 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.331627    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:48:40.427442 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.427399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8813db0-1163-4477-850d-22a635b8a072-tls-certs\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.427635 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.427457    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-model-cache\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.427635 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.427508    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnn7\" (UniqueName: \"kubernetes.io/projected/d8813db0-1163-4477-850d-22a635b8a072-kube-api-access-qvnn7\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.427635 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.427591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-home\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.427635 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.427611    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-dshm\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.427778 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.427638    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-tmp-dir\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.528949 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.528919    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-home\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.528959    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-dshm\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.528986    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-tmp-dir\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529122 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.529103    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8813db0-1163-4477-850d-22a635b8a072-tls-certs\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529346 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.529150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-model-cache\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529346 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.529194    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnn7\" (UniqueName: \"kubernetes.io/projected/d8813db0-1163-4477-850d-22a635b8a072-kube-api-access-qvnn7\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529456 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.529410    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-home\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529456 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.529452    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-tmp-dir\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.529647 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.529627    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-model-cache\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.532092 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.532067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-dshm\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.532315 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.532298    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8813db0-1163-4477-850d-22a635b8a072-tls-certs\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.540464 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.540438    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnn7\" (UniqueName: \"kubernetes.io/projected/d8813db0-1163-4477-850d-22a635b8a072-kube-api-access-qvnn7\") pod \"e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") " pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.573081 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.573040    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:40.715506 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.715476    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"]
Jul 27 19:48:40.718481 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:48:40.718445    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8813db0_1163_4477_850d_22a635b8a072.slice/crio-d45d555d7c4783c1394753c674e10f47dd2c956470c5d2ee18ce5a29e9695417 WatchSource:0}: Error finding container d45d555d7c4783c1394753c674e10f47dd2c956470c5d2ee18ce5a29e9695417: Status 404 returned error can't find the container with id d45d555d7c4783c1394753c674e10f47dd2c956470c5d2ee18ce5a29e9695417
Jul 27 19:48:40.962790 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.962697    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl" event={"ID":"d8813db0-1163-4477-850d-22a635b8a072","Type":"ContainerStarted","Data":"2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64"}
Jul 27 19:48:40.962790 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.962739    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl" event={"ID":"d8813db0-1163-4477-850d-22a635b8a072","Type":"ContainerStarted","Data":"d45d555d7c4783c1394753c674e10f47dd2c956470c5d2ee18ce5a29e9695417"}
Jul 27 19:48:40.985252 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:40.985194    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl" podStartSLOduration=0.98517431 podStartE2EDuration="985.17431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:48:40.983517491 +0000 UTC m=+1469.979365469" watchObservedRunningTime="2026-07-27 19:48:40.98517431 +0000 UTC m=+1469.981022287"
Jul 27 19:48:48.449468 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.449431    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"]
Jul 27 19:48:48.452778 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.452760    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.455668 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.455644    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5a1da6e4-kserve-self-signed-certs\""
Jul 27 19:48:48.465222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.465200    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"]
Jul 27 19:48:48.476017 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.475988    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:48:48.497968 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.497933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-dshm\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.498142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.497975    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25351c11-c62f-479e-ab14-e793ddf24553-tls-certs\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.498142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.498001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frk7m\" (UniqueName: \"kubernetes.io/projected/25351c11-c62f-479e-ab14-e793ddf24553-kube-api-access-frk7m\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.498142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.498061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-model-cache\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.498142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.498093    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-tmp-dir\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.498301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.498150    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-home\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.598904 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.598859    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-dshm\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.598904 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.598907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25351c11-c62f-479e-ab14-e793ddf24553-tls-certs\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599169 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.598925    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frk7m\" (UniqueName: \"kubernetes.io/projected/25351c11-c62f-479e-ab14-e793ddf24553-kube-api-access-frk7m\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599169 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.598947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-model-cache\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599169 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.598962    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-tmp-dir\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599169 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.599006    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-home\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599386 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.599354    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-model-cache\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599444 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.599420    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-home\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.599488 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.599467    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-tmp-dir\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.601290 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.601270    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-dshm\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.601600 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.601581    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25351c11-c62f-479e-ab14-e793ddf24553-tls-certs\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.609301 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.609278    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frk7m\" (UniqueName: \"kubernetes.io/projected/25351c11-c62f-479e-ab14-e793ddf24553-kube-api-access-frk7m\") pod \"e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") " pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.762662 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.762577    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:48.900373 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:48.900340    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"]
Jul 27 19:48:48.904167 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:48:48.904143    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25351c11_c62f_479e_ab14_e793ddf24553.slice/crio-55c3db897e55922579c46777b4111e876b65f66197a1dc4693248d5f10aa84a6 WatchSource:0}: Error finding container 55c3db897e55922579c46777b4111e876b65f66197a1dc4693248d5f10aa84a6: Status 404 returned error can't find the container with id 55c3db897e55922579c46777b4111e876b65f66197a1dc4693248d5f10aa84a6
Jul 27 19:48:49.000469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:49.000441    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w" event={"ID":"25351c11-c62f-479e-ab14-e793ddf24553","Type":"ContainerStarted","Data":"88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36"}
Jul 27 19:48:49.000657 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:49.000478    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w" event={"ID":"25351c11-c62f-479e-ab14-e793ddf24553","Type":"ContainerStarted","Data":"55c3db897e55922579c46777b4111e876b65f66197a1dc4693248d5f10aa84a6"}
Jul 27 19:48:49.020781 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:49.020662    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w" podStartSLOduration=1.020639743 podStartE2EDuration="1.020639743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:48:49.019487074 +0000 UTC m=+1478.015335064" watchObservedRunningTime="2026-07-27 19:48:49.020639743 +0000 UTC m=+1478.016487721"
Jul 27 19:48:50.573308 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:50.573262    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:50.573798 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:50.573493    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:50.585961 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:50.585932    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:51.021235 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:51.021204    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:48:58.762680 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:58.762643    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:58.763071 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:58.762697    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:58.775643 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:58.775617    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:48:59.057568 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:48:59.057538    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:49:10.051732 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:10.051697    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"]
Jul 27 19:49:10.070406 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:10.070373    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:49:10.582045 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:10.582013    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"]
Jul 27 19:49:10.594905 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:10.594882    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:49:11.492089 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:11.492056    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:49:11.492089 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:11.492080    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:49:25.070882 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.070840    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl" podUID="d8813db0-1163-4477-850d-22a635b8a072" containerName="main" containerID="cri-o://2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64" gracePeriod=45
Jul 27 19:49:25.318849 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.318827    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:49:25.434200 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434119    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-model-cache\") pod \"d8813db0-1163-4477-850d-22a635b8a072\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") "
Jul 27 19:49:25.434200 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434166    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-tmp-dir\") pod \"d8813db0-1163-4477-850d-22a635b8a072\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") "
Jul 27 19:49:25.434392 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434212    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8813db0-1163-4477-850d-22a635b8a072-tls-certs\") pod \"d8813db0-1163-4477-850d-22a635b8a072\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") "
Jul 27 19:49:25.434392 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434227    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-home\") pod \"d8813db0-1163-4477-850d-22a635b8a072\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") "
Jul 27 19:49:25.434484 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434421    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvnn7\" (UniqueName: \"kubernetes.io/projected/d8813db0-1163-4477-850d-22a635b8a072-kube-api-access-qvnn7\") pod \"d8813db0-1163-4477-850d-22a635b8a072\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") "
Jul 27 19:49:25.434484 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434454    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8813db0-1163-4477-850d-22a635b8a072" (UID: "d8813db0-1163-4477-850d-22a635b8a072"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.434484 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434459    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8813db0-1163-4477-850d-22a635b8a072" (UID: "d8813db0-1163-4477-850d-22a635b8a072"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.434629 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434483    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-dshm\") pod \"d8813db0-1163-4477-850d-22a635b8a072\" (UID: \"d8813db0-1163-4477-850d-22a635b8a072\") "
Jul 27 19:49:25.434629 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434503    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-home" (OuterVolumeSpecName: "home") pod "d8813db0-1163-4477-850d-22a635b8a072" (UID: "d8813db0-1163-4477-850d-22a635b8a072"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.434827 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434811    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.434895 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434832    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.434895 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.434844    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.436776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.436748    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8813db0-1163-4477-850d-22a635b8a072-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8813db0-1163-4477-850d-22a635b8a072" (UID: "d8813db0-1163-4477-850d-22a635b8a072"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:49:25.436875 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.436786    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-dshm" (OuterVolumeSpecName: "dshm") pod "d8813db0-1163-4477-850d-22a635b8a072" (UID: "d8813db0-1163-4477-850d-22a635b8a072"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.436875 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.436804    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8813db0-1163-4477-850d-22a635b8a072-kube-api-access-qvnn7" (OuterVolumeSpecName: "kube-api-access-qvnn7") pod "d8813db0-1163-4477-850d-22a635b8a072" (UID: "d8813db0-1163-4477-850d-22a635b8a072"). InnerVolumeSpecName "kube-api-access-qvnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:49:25.535987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.535937    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8813db0-1163-4477-850d-22a635b8a072-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.535987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.535983    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvnn7\" (UniqueName: \"kubernetes.io/projected/d8813db0-1163-4477-850d-22a635b8a072-kube-api-access-qvnn7\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.535987 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.535995    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8813db0-1163-4477-850d-22a635b8a072-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.601223 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.601187    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w" podUID="25351c11-c62f-479e-ab14-e793ddf24553" containerName="main" containerID="cri-o://88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36" gracePeriod=45
Jul 27 19:49:25.843649 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.843627    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:49:25.939805 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.939772    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-home\") pod \"25351c11-c62f-479e-ab14-e793ddf24553\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") "
Jul 27 19:49:25.939999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.939824    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-model-cache\") pod \"25351c11-c62f-479e-ab14-e793ddf24553\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") "
Jul 27 19:49:25.939999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.939854    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-tmp-dir\") pod \"25351c11-c62f-479e-ab14-e793ddf24553\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") "
Jul 27 19:49:25.939999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.939938    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-dshm\") pod \"25351c11-c62f-479e-ab14-e793ddf24553\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") "
Jul 27 19:49:25.939999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.939965    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25351c11-c62f-479e-ab14-e793ddf24553-tls-certs\") pod \"25351c11-c62f-479e-ab14-e793ddf24553\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") "
Jul 27 19:49:25.940211 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940022    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frk7m\" (UniqueName: \"kubernetes.io/projected/25351c11-c62f-479e-ab14-e793ddf24553-kube-api-access-frk7m\") pod \"25351c11-c62f-479e-ab14-e793ddf24553\" (UID: \"25351c11-c62f-479e-ab14-e793ddf24553\") "
Jul 27 19:49:25.940211 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940047    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-home" (OuterVolumeSpecName: "home") pod "25351c11-c62f-479e-ab14-e793ddf24553" (UID: "25351c11-c62f-479e-ab14-e793ddf24553"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.940211 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940102    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-model-cache" (OuterVolumeSpecName: "model-cache") pod "25351c11-c62f-479e-ab14-e793ddf24553" (UID: "25351c11-c62f-479e-ab14-e793ddf24553"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.940211 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940158    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25351c11-c62f-479e-ab14-e793ddf24553" (UID: "25351c11-c62f-479e-ab14-e793ddf24553"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.940385 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940363    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.940440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940392    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.940440 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.940409    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:25.942150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.942121    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-dshm" (OuterVolumeSpecName: "dshm") pod "25351c11-c62f-479e-ab14-e793ddf24553" (UID: "25351c11-c62f-479e-ab14-e793ddf24553"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:49:25.942150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.942133    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25351c11-c62f-479e-ab14-e793ddf24553-kube-api-access-frk7m" (OuterVolumeSpecName: "kube-api-access-frk7m") pod "25351c11-c62f-479e-ab14-e793ddf24553" (UID: "25351c11-c62f-479e-ab14-e793ddf24553"). InnerVolumeSpecName "kube-api-access-frk7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:49:25.942293 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:25.942174    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25351c11-c62f-479e-ab14-e793ddf24553-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25351c11-c62f-479e-ab14-e793ddf24553" (UID: "25351c11-c62f-479e-ab14-e793ddf24553"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:49:26.040862 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.040833    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25351c11-c62f-479e-ab14-e793ddf24553-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:26.040862 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.040858    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25351c11-c62f-479e-ab14-e793ddf24553-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:26.041091 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.040878    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frk7m\" (UniqueName: \"kubernetes.io/projected/25351c11-c62f-479e-ab14-e793ddf24553-kube-api-access-frk7m\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:49:26.175716 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.175681    2568 generic.go:356] "Generic (PLEG): container finished" podID="25351c11-c62f-479e-ab14-e793ddf24553" containerID="88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36" exitCode=0
Jul 27 19:49:26.176164 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.175726    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"
Jul 27 19:49:26.176164 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.175725    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w" event={"ID":"25351c11-c62f-479e-ab14-e793ddf24553","Type":"ContainerDied","Data":"88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36"}
Jul 27 19:49:26.176164 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.175771    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w" event={"ID":"25351c11-c62f-479e-ab14-e793ddf24553","Type":"ContainerDied","Data":"55c3db897e55922579c46777b4111e876b65f66197a1dc4693248d5f10aa84a6"}
Jul 27 19:49:26.176164 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.175793    2568 scope.go:117] "RemoveContainer" containerID="88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36"
Jul 27 19:49:26.177326 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.177235    2568 generic.go:356] "Generic (PLEG): container finished" podID="d8813db0-1163-4477-850d-22a635b8a072" containerID="2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64" exitCode=0
Jul 27 19:49:26.177326 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.177268    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"
Jul 27 19:49:26.177430 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.177268    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl" event={"ID":"d8813db0-1163-4477-850d-22a635b8a072","Type":"ContainerDied","Data":"2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64"}
Jul 27 19:49:26.177430 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.177367    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl" event={"ID":"d8813db0-1163-4477-850d-22a635b8a072","Type":"ContainerDied","Data":"d45d555d7c4783c1394753c674e10f47dd2c956470c5d2ee18ce5a29e9695417"}
Jul 27 19:49:26.185409 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.185386    2568 scope.go:117] "RemoveContainer" containerID="88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36"
Jul 27 19:49:26.185691 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:49:26.185666    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36\": container with ID starting with 88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36 not found: ID does not exist" containerID="88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36"
Jul 27 19:49:26.185751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.185704    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36"} err="failed to get container status \"88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36\": rpc error: code = NotFound desc = could not find container \"88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36\": container with ID starting with 88f0471f7fe8a1f21e756f2e2332d09a8e7f051197aed52b4527164427870c36 not found: ID does not exist"
Jul 27 19:49:26.185751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.185727    2568 scope.go:117] "RemoveContainer" containerID="2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64"
Jul 27 19:49:26.193142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.193127    2568 scope.go:117] "RemoveContainer" containerID="2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64"
Jul 27 19:49:26.193398 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:49:26.193381    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64\": container with ID starting with 2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64 not found: ID does not exist" containerID="2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64"
Jul 27 19:49:26.193445 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.193404    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64"} err="failed to get container status \"2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64\": rpc error: code = NotFound desc = could not find container \"2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64\": container with ID starting with 2b131e025e87c1793eb3d7d69931ea253b524ac58e11f3d85303debc7a758f64 not found: ID does not exist"
Jul 27 19:49:26.197784 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.197765    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"]
Jul 27 19:49:26.203689 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.203671    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5a1da6e4-kserve-675b679b59-2vqxl"]
Jul 27 19:49:26.218757 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.218735    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"]
Jul 27 19:49:26.226942 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:26.224323    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5a1da6e4-kserve-5b5f596b9f-6dv4w"]
Jul 27 19:49:27.567769 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:27.567734    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25351c11-c62f-479e-ab14-e793ddf24553" path="/var/lib/kubelet/pods/25351c11-c62f-479e-ab14-e793ddf24553/volumes"
Jul 27 19:49:27.568139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:49:27.568125    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8813db0-1163-4477-850d-22a635b8a072" path="/var/lib/kubelet/pods/d8813db0-1163-4477-850d-22a635b8a072/volumes"
Jul 27 19:50:04.908047 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:50:04.908011    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:50:05.009243 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:50:05.009208    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:18.899708 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:18.899677    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:26.213776 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:26.213744    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:35.381500 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:35.381391    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:44.490687 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:44.490650    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:47.729930 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:47.729897    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:47.823263 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:47.823227    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:48.182663 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:48.182626    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:55.177023 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:55.176966    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:55.252662 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:55.252631    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:55.420466 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:55.420419    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:55.832153 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:55.832122    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:55.947087 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:55.947058    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:58.571938 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.571901    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:58.673730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.673697    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:58.687217 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687190    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"]
Jul 27 19:52:58.687552 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687540    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8813db0-1163-4477-850d-22a635b8a072" containerName="main"
Jul 27 19:52:58.687603 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687555    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8813db0-1163-4477-850d-22a635b8a072" containerName="main"
Jul 27 19:52:58.687603 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687563    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25351c11-c62f-479e-ab14-e793ddf24553" containerName="main"
Jul 27 19:52:58.687603 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687569    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="25351c11-c62f-479e-ab14-e793ddf24553" containerName="main"
Jul 27 19:52:58.687691 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687636    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="25351c11-c62f-479e-ab14-e793ddf24553" containerName="main"
Jul 27 19:52:58.687691 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.687646    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8813db0-1163-4477-850d-22a635b8a072" containerName="main"
Jul 27 19:52:58.690857 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.690840    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.693863 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.693843    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e5b0400d-serving-cert\""
Jul 27 19:52:58.703172 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.703141    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz5l\" (UniqueName: \"kubernetes.io/projected/3b90b70d-9794-4174-ba29-32521c25b329-kube-api-access-4vz5l\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.703270 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.703192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b90b70d-9794-4174-ba29-32521c25b329-maas-api-tls\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.703270 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.703252    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b90b70d-9794-4174-ba29-32521c25b329-openshift-service-ca\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.709290 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.709267    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"]
Jul 27 19:52:58.775137 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.775106    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:58.804077 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.804043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b90b70d-9794-4174-ba29-32521c25b329-openshift-service-ca\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.804296 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.804100    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz5l\" (UniqueName: \"kubernetes.io/projected/3b90b70d-9794-4174-ba29-32521c25b329-kube-api-access-4vz5l\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.804296 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.804140    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b90b70d-9794-4174-ba29-32521c25b329-maas-api-tls\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.804835 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.804811    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b90b70d-9794-4174-ba29-32521c25b329-openshift-service-ca\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.806768 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.806738    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b90b70d-9794-4174-ba29-32521c25b329-maas-api-tls\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.815947 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.815923    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz5l\" (UniqueName: \"kubernetes.io/projected/3b90b70d-9794-4174-ba29-32521c25b329-kube-api-access-4vz5l\") pod \"maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:58.835927 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:58.835838    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:59.000354 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.000318    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:52:59.146433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.146409    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"]
Jul 27 19:52:59.149138 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:52:59.149110    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b90b70d_9794_4174_ba29_32521c25b329.slice/crio-06199e1c534b3b96628396bf19ea578452993c00e480892535cdca8d5b89cfd9 WatchSource:0}: Error finding container 06199e1c534b3b96628396bf19ea578452993c00e480892535cdca8d5b89cfd9: Status 404 returned error can't find the container with id 06199e1c534b3b96628396bf19ea578452993c00e480892535cdca8d5b89cfd9
Jul 27 19:52:59.150391 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.150370    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:52:59.401346 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.401266    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:59.504908 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.504878    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:52:59.624931 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.624892    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"]
Jul 27 19:52:59.629433 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.629414    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.632906 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.632881    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e5b0400d\""
Jul 27 19:52:59.633131 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.633113    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e5b0400d-dockercfg-6xw8x\""
Jul 27 19:52:59.641861 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.641840    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"]
Jul 27 19:52:59.708874 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.708793    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"]
Jul 27 19:52:59.709888 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.709861    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhdzc\" (UniqueName: \"kubernetes.io/projected/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-kube-api-access-lhdzc\") pod \"payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.710117 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.710080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.712589 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.712571    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.726293 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.726267    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"]
Jul 27 19:52:59.810493 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.810464    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13d8a42-a847-425b-90de-fce05452d40e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") " pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.810682 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.810501    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9x7\" (UniqueName: \"kubernetes.io/projected/f13d8a42-a847-425b-90de-fce05452d40e-kube-api-access-pv9x7\") pod \"payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") " pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.810682 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.810605    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.810682 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.810635    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhdzc\" (UniqueName: \"kubernetes.io/projected/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-kube-api-access-lhdzc\") pod \"payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.811242 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.811225    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.830354 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.830328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhdzc\" (UniqueName: \"kubernetes.io/projected/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-kube-api-access-lhdzc\") pod \"payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:52:59.912082 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.912058    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13d8a42-a847-425b-90de-fce05452d40e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") " pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.912174 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.912101    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9x7\" (UniqueName: \"kubernetes.io/projected/f13d8a42-a847-425b-90de-fce05452d40e-kube-api-access-pv9x7\") pod \"payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") " pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.912732 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.912715    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13d8a42-a847-425b-90de-fce05452d40e-plugins-config-volume\") pod \"payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") " pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.923150 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.923126    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9x7\" (UniqueName: \"kubernetes.io/projected/f13d8a42-a847-425b-90de-fce05452d40e-kube-api-access-pv9x7\") pod \"payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") " pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:52:59.941006 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:52:59.940985    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:53:00.023833 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.023799    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:53:00.097299 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.097274    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"]
Jul 27 19:53:00.099272 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:53:00.099246    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf73465_b35c_4f0d_9299_7f786f4c6cd1.slice/crio-7307904fd6a8e2d46f11338683015d439aae60b9874e481108658d57e8c9a026 WatchSource:0}: Error finding container 7307904fd6a8e2d46f11338683015d439aae60b9874e481108658d57e8c9a026: Status 404 returned error can't find the container with id 7307904fd6a8e2d46f11338683015d439aae60b9874e481108658d57e8c9a026
Jul 27 19:53:00.136093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.135967    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj" event={"ID":"3b90b70d-9794-4174-ba29-32521c25b329","Type":"ContainerStarted","Data":"314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4"}
Jul 27 19:53:00.136093 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.136030    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj" event={"ID":"3b90b70d-9794-4174-ba29-32521c25b329","Type":"ContainerStarted","Data":"06199e1c534b3b96628396bf19ea578452993c00e480892535cdca8d5b89cfd9"}
Jul 27 19:53:00.136320 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.136191    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:53:00.137588 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.137546    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w" event={"ID":"fdf73465-b35c-4f0d-9299-7f786f4c6cd1","Type":"ContainerStarted","Data":"7307904fd6a8e2d46f11338683015d439aae60b9874e481108658d57e8c9a026"}
Jul 27 19:53:00.194263 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.194207    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj" podStartSLOduration=1.49887796 podStartE2EDuration="2.194188883s" totalImagesPullingTime="695.310923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:53:00.18878556 +0000 UTC m=+1729.184633539" watchObservedRunningTime="2026-07-27 19:53:00.194188883 +0000 UTC m=+1729.190036860"
Jul 27 19:53:00.197516 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:00.197492    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"]
Jul 27 19:53:00.197791 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:53:00.197642    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13d8a42_a847_425b_90de_fce05452d40e.slice/crio-59cb0ae0915b06f8a20f9cf83614ee8c1d9d61b3b8f1a85fa30732bd729b6db8 WatchSource:0}: Error finding container 59cb0ae0915b06f8a20f9cf83614ee8c1d9d61b3b8f1a85fa30732bd729b6db8: Status 404 returned error can't find the container with id 59cb0ae0915b06f8a20f9cf83614ee8c1d9d61b3b8f1a85fa30732bd729b6db8
Jul 27 19:53:01.143426 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:01.143393    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64" event={"ID":"f13d8a42-a847-425b-90de-fce05452d40e","Type":"ContainerStarted","Data":"3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160"}
Jul 27 19:53:01.143922 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:01.143456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64" event={"ID":"f13d8a42-a847-425b-90de-fce05452d40e","Type":"ContainerStarted","Data":"59cb0ae0915b06f8a20f9cf83614ee8c1d9d61b3b8f1a85fa30732bd729b6db8"}
Jul 27 19:53:01.143922 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:01.143491    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:53:01.144860 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:01.144833    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w" event={"ID":"fdf73465-b35c-4f0d-9299-7f786f4c6cd1","Type":"ContainerStarted","Data":"8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9"}
Jul 27 19:53:01.166406 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:01.166328    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64" podStartSLOduration=2.166314254 podStartE2EDuration="2.166314254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:53:01.165166952 +0000 UTC m=+1730.161014939" watchObservedRunningTime="2026-07-27 19:53:01.166314254 +0000 UTC m=+1730.162162230"
Jul 27 19:53:01.194373 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:01.194330    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w" podStartSLOduration=2.19431485 podStartE2EDuration="2.19431485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:53:01.193442444 +0000 UTC m=+1730.189290423" watchObservedRunningTime="2026-07-27 19:53:01.19431485 +0000 UTC m=+1730.190162867"
Jul 27 19:53:02.149543 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:02.149495    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:53:06.150304 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:06.150223    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:53:12.151978 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:12.151948    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:53:13.156100 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:13.156074    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:53:35.675515 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.675477    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7954c54d9b-vbmzh"]
Jul 27 19:53:35.679356 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.679334    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.694792 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.694766    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7954c54d9b-vbmzh"]
Jul 27 19:53:35.739081 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.739035    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:53:35.820513 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.820479    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:53:35.833622 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833588    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-service-ca\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.833705 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-oauth-serving-cert\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.833705 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833665    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-trusted-ca-bundle\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.833774 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833731    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-serving-cert\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.833813 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833794    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-oauth-config\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.833858 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833846    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vbc\" (UniqueName: \"kubernetes.io/projected/1b73b0c8-3b97-4d78-a170-5e084f5be495-kube-api-access-g4vbc\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.833897 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.833879    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-config\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.934980    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-config\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935022    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-service-ca\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935065 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935047    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-oauth-serving-cert\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935080    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-trusted-ca-bundle\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935116    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-serving-cert\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-oauth-config\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935239    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vbc\" (UniqueName: \"kubernetes.io/projected/1b73b0c8-3b97-4d78-a170-5e084f5be495-kube-api-access-g4vbc\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935925 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935895    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-service-ca\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.935925 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935922    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-oauth-serving-cert\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.936134 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.935924    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-config\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.936134 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.936122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b73b0c8-3b97-4d78-a170-5e084f5be495-trusted-ca-bundle\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.937821 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.937789    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-oauth-config\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.937920 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.937859    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b73b0c8-3b97-4d78-a170-5e084f5be495-console-serving-cert\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.946143 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.946124    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vbc\" (UniqueName: \"kubernetes.io/projected/1b73b0c8-3b97-4d78-a170-5e084f5be495-kube-api-access-g4vbc\") pod \"console-7954c54d9b-vbmzh\" (UID: \"1b73b0c8-3b97-4d78-a170-5e084f5be495\") " pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:35.989640 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:35.989611    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:36.133351 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:36.133327    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7954c54d9b-vbmzh"]
Jul 27 19:53:36.135147 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:53:36.135121    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b73b0c8_3b97_4d78_a170_5e084f5be495.slice/crio-3625f37a353592795a4bd290c0ee2d54f4cd5266e1a09d03cef8dcd5e9140490 WatchSource:0}: Error finding container 3625f37a353592795a4bd290c0ee2d54f4cd5266e1a09d03cef8dcd5e9140490: Status 404 returned error can't find the container with id 3625f37a353592795a4bd290c0ee2d54f4cd5266e1a09d03cef8dcd5e9140490
Jul 27 19:53:36.314864 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:36.314815    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7954c54d9b-vbmzh" event={"ID":"1b73b0c8-3b97-4d78-a170-5e084f5be495","Type":"ContainerStarted","Data":"b684482dbc2e3c27676fd5d2331691782a43dd33f7df68e567dc92aad747c86a"}
Jul 27 19:53:36.315058 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:36.314875    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7954c54d9b-vbmzh" event={"ID":"1b73b0c8-3b97-4d78-a170-5e084f5be495","Type":"ContainerStarted","Data":"3625f37a353592795a4bd290c0ee2d54f4cd5266e1a09d03cef8dcd5e9140490"}
Jul 27 19:53:36.339000 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:36.338950    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7954c54d9b-vbmzh" podStartSLOduration=1.338933939 podStartE2EDuration="1.338933939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:53:36.338322779 +0000 UTC m=+1765.334170759" watchObservedRunningTime="2026-07-27 19:53:36.338933939 +0000 UTC m=+1765.334781916"
Jul 27 19:53:45.990603 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:45.990564    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:45.991002 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:45.990630    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:45.995396 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:45.995369    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:46.367183 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:46.367155    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7954c54d9b-vbmzh"
Jul 27 19:53:46.427004 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:46.426972    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4754cf4-4qcf6"]
Jul 27 19:53:46.566839 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:46.566808    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:53:46.663868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:46.663763    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:53:46.766810 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:53:46.766778    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:54:11.450322 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.450274    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56b4754cf4-4qcf6" podUID="dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" containerName="console" containerID="cri-o://cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00" gracePeriod=15
Jul 27 19:54:11.525434 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.525405    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:54:11.526993 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.526969    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:54:11.712252 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.712184    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:54:11.873137 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873102    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-oauth-serving-cert\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873147    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-oauth-config\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873188    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwjn9\" (UniqueName: \"kubernetes.io/projected/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-kube-api-access-hwjn9\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873251    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-service-ca\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873276    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-serving-cert\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873576 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873443    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-config\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873576 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873486    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:54:11.873576 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873517    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-trusted-ca-bundle\") pod \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\" (UID: \"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357\") "
Jul 27 19:54:11.873883 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873764    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:54:11.873883 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873830    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:54:11.874215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873901    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:11.874215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.873920    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-oauth-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:11.874215 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.874160    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-config" (OuterVolumeSpecName: "console-config") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:54:11.875823 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.875797    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:54:11.875823 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.875804    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:54:11.875988 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.875881    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-kube-api-access-hwjn9" (OuterVolumeSpecName: "kube-api-access-hwjn9") pod "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" (UID: "dc35cdf4-07f6-49ac-9a7f-5c078e8b9357"). InnerVolumeSpecName "kube-api-access-hwjn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:54:11.974469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.974370    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjn9\" (UniqueName: \"kubernetes.io/projected/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-kube-api-access-hwjn9\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:11.974469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.974414    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-serving-cert\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:11.974469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.974426    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:11.974469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.974434    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-trusted-ca-bundle\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:11.974469 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:11.974442    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357-console-oauth-config\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:54:12.483828 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.483796    2568 generic.go:356] "Generic (PLEG): container finished" podID="dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" containerID="cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00" exitCode=0
Jul 27 19:54:12.484236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.483845    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4754cf4-4qcf6"
Jul 27 19:54:12.484236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.483892    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4754cf4-4qcf6" event={"ID":"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357","Type":"ContainerDied","Data":"cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00"}
Jul 27 19:54:12.484236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.483938    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4754cf4-4qcf6" event={"ID":"dc35cdf4-07f6-49ac-9a7f-5c078e8b9357","Type":"ContainerDied","Data":"9903629160d8d9531458bef55b410e542b3020c49248f140c18e1b76421c7b15"}
Jul 27 19:54:12.484236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.483957    2568 scope.go:117] "RemoveContainer" containerID="cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00"
Jul 27 19:54:12.493370 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.493348    2568 scope.go:117] "RemoveContainer" containerID="cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00"
Jul 27 19:54:12.493664 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:54:12.493644    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00\": container with ID starting with cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00 not found: ID does not exist" containerID="cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00"
Jul 27 19:54:12.493747 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.493671    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00"} err="failed to get container status \"cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00\": rpc error: code = NotFound desc = could not find container \"cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00\": container with ID starting with cc2fe91a8cfa897c9f66685b581eef5ed41dc9213ba41173c644af0b0e2b1b00 not found: ID does not exist"
Jul 27 19:54:12.510751 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.510729    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4754cf4-4qcf6"]
Jul 27 19:54:12.513964 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:12.513942    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56b4754cf4-4qcf6"]
Jul 27 19:54:13.568729 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:54:13.568695    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" path="/var/lib/kubelet/pods/dc35cdf4-07f6-49ac-9a7f-5c078e8b9357/volumes"
Jul 27 19:55:19.076382 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:55:19.076349    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:05.209730 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.209694    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"]
Jul 27 19:58:05.210198 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.210063    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" containerName="console"
Jul 27 19:58:05.210198 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.210075    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" containerName="console"
Jul 27 19:58:05.210198 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.210154    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc35cdf4-07f6-49ac-9a7f-5c078e8b9357" containerName="console"
Jul 27 19:58:05.213054 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.213038    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:05.215827 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.215808    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 19:58:05.228814 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.228788    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"]
Jul 27 19:58:05.312424 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.312385    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16699063-3cd1-4c81-87ec-a8d2497db12a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2\" (UID: \"16699063-3cd1-4c81-87ec-a8d2497db12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:05.413910 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.413872    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16699063-3cd1-4c81-87ec-a8d2497db12a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2\" (UID: \"16699063-3cd1-4c81-87ec-a8d2497db12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:05.414508 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.414490    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16699063-3cd1-4c81-87ec-a8d2497db12a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2\" (UID: \"16699063-3cd1-4c81-87ec-a8d2497db12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:05.522979 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.522951    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:05.661485 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.661458    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"]
Jul 27 19:58:05.664116 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:58:05.664086    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16699063_3cd1_4c81_87ec_a8d2497db12a.slice/crio-70bf48bef8954eacb265f19c440d863e8dd6ad8dad384b1ec2ac28b7c00e4e49 WatchSource:0}: Error finding container 70bf48bef8954eacb265f19c440d863e8dd6ad8dad384b1ec2ac28b7c00e4e49: Status 404 returned error can't find the container with id 70bf48bef8954eacb265f19c440d863e8dd6ad8dad384b1ec2ac28b7c00e4e49
Jul 27 19:58:05.666167 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:05.666150    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:58:06.566332 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:06.566294    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2" event={"ID":"16699063-3cd1-4c81-87ec-a8d2497db12a","Type":"ContainerStarted","Data":"70bf48bef8954eacb265f19c440d863e8dd6ad8dad384b1ec2ac28b7c00e4e49"}
Jul 27 19:58:09.581099 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:09.581064    2568 generic.go:356] "Generic (PLEG): container finished" podID="16699063-3cd1-4c81-87ec-a8d2497db12a" containerID="1ee3b18281bf1a433f956766ad5599aa158e2bcc104f75fd41718c8569bced88" exitCode=0
Jul 27 19:58:09.581484 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:09.581123    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2" event={"ID":"16699063-3cd1-4c81-87ec-a8d2497db12a","Type":"ContainerDied","Data":"1ee3b18281bf1a433f956766ad5599aa158e2bcc104f75fd41718c8569bced88"}
Jul 27 19:58:10.716786 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:10.716762    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:10.762567 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:10.762519    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16699063-3cd1-4c81-87ec-a8d2497db12a-maas-api-service-ca\") pod \"16699063-3cd1-4c81-87ec-a8d2497db12a\" (UID: \"16699063-3cd1-4c81-87ec-a8d2497db12a\") "
Jul 27 19:58:10.762850 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:10.762830    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16699063-3cd1-4c81-87ec-a8d2497db12a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16699063-3cd1-4c81-87ec-a8d2497db12a" (UID: "16699063-3cd1-4c81-87ec-a8d2497db12a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:58:10.863279 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:10.863197    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16699063-3cd1-4c81-87ec-a8d2497db12a-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:11.592034 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:11.591995    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2" event={"ID":"16699063-3cd1-4c81-87ec-a8d2497db12a","Type":"ContainerDied","Data":"70bf48bef8954eacb265f19c440d863e8dd6ad8dad384b1ec2ac28b7c00e4e49"}
Jul 27 19:58:11.592166 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:11.592035    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"
Jul 27 19:58:11.592166 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:11.592048    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70bf48bef8954eacb265f19c440d863e8dd6ad8dad384b1ec2ac28b7c00e4e49"
Jul 27 19:58:15.714035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.714000    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"]
Jul 27 19:58:15.714435 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.714187    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj" podUID="3b90b70d-9794-4174-ba29-32521c25b329" containerName="maas-api" containerID="cri-o://314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4" gracePeriod=30
Jul 27 19:58:15.761808 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.761783    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:15.778019 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.777983    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"]
Jul 27 19:58:15.778254 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.778231    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64" podUID="f13d8a42-a847-425b-90de-fce05452d40e" containerName="payload-processing" containerID="cri-o://3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160" gracePeriod=30
Jul 27 19:58:15.793538 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.793502    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"]
Jul 27 19:58:15.794402 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:15.793930    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w" podUID="fdf73465-b35c-4f0d-9299-7f786f4c6cd1" containerName="payload-pre-processing" containerID="cri-o://8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9" gracePeriod=30
Jul 27 19:58:16.063963 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.063938    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:58:16.087655 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.087631    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:58:16.099168 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.099147    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:58:16.111371 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.111346    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b90b70d-9794-4174-ba29-32521c25b329-maas-api-tls\") pod \"3b90b70d-9794-4174-ba29-32521c25b329\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") "
Jul 27 19:58:16.111563 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.111378    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b90b70d-9794-4174-ba29-32521c25b329-openshift-service-ca\") pod \"3b90b70d-9794-4174-ba29-32521c25b329\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") "
Jul 27 19:58:16.111563 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.111413    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vz5l\" (UniqueName: \"kubernetes.io/projected/3b90b70d-9794-4174-ba29-32521c25b329-kube-api-access-4vz5l\") pod \"3b90b70d-9794-4174-ba29-32521c25b329\" (UID: \"3b90b70d-9794-4174-ba29-32521c25b329\") "
Jul 27 19:58:16.111792 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.111764    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b90b70d-9794-4174-ba29-32521c25b329-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b90b70d-9794-4174-ba29-32521c25b329" (UID: "3b90b70d-9794-4174-ba29-32521c25b329"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:58:16.113999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.113974    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b90b70d-9794-4174-ba29-32521c25b329-kube-api-access-4vz5l" (OuterVolumeSpecName: "kube-api-access-4vz5l") pod "3b90b70d-9794-4174-ba29-32521c25b329" (UID: "3b90b70d-9794-4174-ba29-32521c25b329"). InnerVolumeSpecName "kube-api-access-4vz5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:58:16.113999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.113984    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b90b70d-9794-4174-ba29-32521c25b329-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b90b70d-9794-4174-ba29-32521c25b329" (UID: "3b90b70d-9794-4174-ba29-32521c25b329"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:58:16.212794 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.212764    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhdzc\" (UniqueName: \"kubernetes.io/projected/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-kube-api-access-lhdzc\") pod \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") "
Jul 27 19:58:16.212966 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.212839    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13d8a42-a847-425b-90de-fce05452d40e-plugins-config-volume\") pod \"f13d8a42-a847-425b-90de-fce05452d40e\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") "
Jul 27 19:58:16.212966 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.212880    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-plugins-config-volume\") pod \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\" (UID: \"fdf73465-b35c-4f0d-9299-7f786f4c6cd1\") "
Jul 27 19:58:16.212966 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.212912    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv9x7\" (UniqueName: \"kubernetes.io/projected/f13d8a42-a847-425b-90de-fce05452d40e-kube-api-access-pv9x7\") pod \"f13d8a42-a847-425b-90de-fce05452d40e\" (UID: \"f13d8a42-a847-425b-90de-fce05452d40e\") "
Jul 27 19:58:16.213158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.213051    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b90b70d-9794-4174-ba29-32521c25b329-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.213158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.213061    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b90b70d-9794-4174-ba29-32521c25b329-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.213158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.213070    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vz5l\" (UniqueName: \"kubernetes.io/projected/3b90b70d-9794-4174-ba29-32521c25b329-kube-api-access-4vz5l\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.213309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.213242    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdf73465-b35c-4f0d-9299-7f786f4c6cd1" (UID: "fdf73465-b35c-4f0d-9299-7f786f4c6cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:58:16.213309 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.213288    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13d8a42-a847-425b-90de-fce05452d40e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f13d8a42-a847-425b-90de-fce05452d40e" (UID: "f13d8a42-a847-425b-90de-fce05452d40e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 19:58:16.215238 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.215217    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13d8a42-a847-425b-90de-fce05452d40e-kube-api-access-pv9x7" (OuterVolumeSpecName: "kube-api-access-pv9x7") pod "f13d8a42-a847-425b-90de-fce05452d40e" (UID: "f13d8a42-a847-425b-90de-fce05452d40e"). InnerVolumeSpecName "kube-api-access-pv9x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:58:16.215327 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.215254    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-kube-api-access-lhdzc" (OuterVolumeSpecName: "kube-api-access-lhdzc") pod "fdf73465-b35c-4f0d-9299-7f786f4c6cd1" (UID: "fdf73465-b35c-4f0d-9299-7f786f4c6cd1"). InnerVolumeSpecName "kube-api-access-lhdzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:58:16.313926 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.313893    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.313926 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.313925    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv9x7\" (UniqueName: \"kubernetes.io/projected/f13d8a42-a847-425b-90de-fce05452d40e-kube-api-access-pv9x7\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.313926 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.313934    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhdzc\" (UniqueName: \"kubernetes.io/projected/fdf73465-b35c-4f0d-9299-7f786f4c6cd1-kube-api-access-lhdzc\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.313926 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.313944    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13d8a42-a847-425b-90de-fce05452d40e-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 19:58:16.616231 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.616139    2568 generic.go:356] "Generic (PLEG): container finished" podID="f13d8a42-a847-425b-90de-fce05452d40e" containerID="3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160" exitCode=0
Jul 27 19:58:16.616231 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.616184    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"
Jul 27 19:58:16.616231 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.616224    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64" event={"ID":"f13d8a42-a847-425b-90de-fce05452d40e","Type":"ContainerDied","Data":"3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160"}
Jul 27 19:58:16.616488 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.616278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64" event={"ID":"f13d8a42-a847-425b-90de-fce05452d40e","Type":"ContainerDied","Data":"59cb0ae0915b06f8a20f9cf83614ee8c1d9d61b3b8f1a85fa30732bd729b6db8"}
Jul 27 19:58:16.616488 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.616308    2568 scope.go:117] "RemoveContainer" containerID="3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160"
Jul 27 19:58:16.617594 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.617566    2568 generic.go:356] "Generic (PLEG): container finished" podID="3b90b70d-9794-4174-ba29-32521c25b329" containerID="314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4" exitCode=0
Jul 27 19:58:16.617712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.617601    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"
Jul 27 19:58:16.617712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.617638    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj" event={"ID":"3b90b70d-9794-4174-ba29-32521c25b329","Type":"ContainerDied","Data":"314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4"}
Jul 27 19:58:16.617712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.617691    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj" event={"ID":"3b90b70d-9794-4174-ba29-32521c25b329","Type":"ContainerDied","Data":"06199e1c534b3b96628396bf19ea578452993c00e480892535cdca8d5b89cfd9"}
Jul 27 19:58:16.619300 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.619279    2568 generic.go:356] "Generic (PLEG): container finished" podID="fdf73465-b35c-4f0d-9299-7f786f4c6cd1" containerID="8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9" exitCode=0
Jul 27 19:58:16.619389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.619356    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"
Jul 27 19:58:16.619389 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.619355    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w" event={"ID":"fdf73465-b35c-4f0d-9299-7f786f4c6cd1","Type":"ContainerDied","Data":"8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9"}
Jul 27 19:58:16.619535 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.619404    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w" event={"ID":"fdf73465-b35c-4f0d-9299-7f786f4c6cd1","Type":"ContainerDied","Data":"7307904fd6a8e2d46f11338683015d439aae60b9874e481108658d57e8c9a026"}
Jul 27 19:58:16.626727 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.626711    2568 scope.go:117] "RemoveContainer" containerID="3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160"
Jul 27 19:58:16.627866 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:58:16.627732    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160\": container with ID starting with 3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160 not found: ID does not exist" containerID="3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160"
Jul 27 19:58:16.627866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.627765    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160"} err="failed to get container status \"3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160\": rpc error: code = NotFound desc = could not find container \"3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160\": container with ID starting with 3f8a8f526e9e7d54dd7ec718d3a13f07b6281f2ed5656d2fef9b5284dafbf160 not found: ID does not exist"
Jul 27 19:58:16.627866 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.627787    2568 scope.go:117] "RemoveContainer" containerID="314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4"
Jul 27 19:58:16.637802 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.637786    2568 scope.go:117] "RemoveContainer" containerID="314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4"
Jul 27 19:58:16.638061 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:58:16.638039    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4\": container with ID starting with 314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4 not found: ID does not exist" containerID="314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4"
Jul 27 19:58:16.638112 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.638072    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4"} err="failed to get container status \"314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4\": rpc error: code = NotFound desc = could not find container \"314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4\": container with ID starting with 314b6489d7f06957f4893ee613f25bba8da20c25aeaa4b05cccc160d893a8eb4 not found: ID does not exist"
Jul 27 19:58:16.638112 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.638094    2568 scope.go:117] "RemoveContainer" containerID="8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9"
Jul 27 19:58:16.645581 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.645552    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"]
Jul 27 19:58:16.646631 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.646601    2568 scope.go:117] "RemoveContainer" containerID="8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9"
Jul 27 19:58:16.646873 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:58:16.646855    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9\": container with ID starting with 8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9 not found: ID does not exist" containerID="8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9"
Jul 27 19:58:16.646923 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.646881    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9"} err="failed to get container status \"8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9\": rpc error: code = NotFound desc = could not find container \"8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9\": container with ID starting with 8380bc21c37838ef164a187cac69ef4c46fe8d77ed8eb33ca95199e83761a6e9 not found: ID does not exist"
Jul 27 19:58:16.648260 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.648242    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e5b0400d-64f4999684-qkn64"]
Jul 27 19:58:16.661868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.661842    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"]
Jul 27 19:58:16.664305 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.664280    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e5b0400d-66d689bcc6-f6lcj"]
Jul 27 19:58:16.677716 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.677686    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"]
Jul 27 19:58:16.682564 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:16.682539    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e5b0400d-7fbf4b9b48-k2r9w"]
Jul 27 19:58:17.100868 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:17.100837    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:17.568805 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:17.568778    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b90b70d-9794-4174-ba29-32521c25b329" path="/var/lib/kubelet/pods/3b90b70d-9794-4174-ba29-32521c25b329/volumes"
Jul 27 19:58:17.569126 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:17.569114    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13d8a42-a847-425b-90de-fce05452d40e" path="/var/lib/kubelet/pods/f13d8a42-a847-425b-90de-fce05452d40e/volumes"
Jul 27 19:58:17.569408 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:17.569397    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf73465-b35c-4f0d-9299-7f786f4c6cd1" path="/var/lib/kubelet/pods/fdf73465-b35c-4f0d-9299-7f786f4c6cd1/volumes"
Jul 27 19:58:20.304741 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:20.304704    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"]
Jul 27 19:58:20.308063 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:20.308035    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e5b0400d-xz6j2"]
Jul 27 19:58:20.326999 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:20.326948    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:20.461946 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:20.461919    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:20.745168 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:20.745056    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:20.974912 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:20.974884    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:21.075024 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:21.074989    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:21.189573 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:21.189540    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:21.523825 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:21.523790    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:21.569142 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:21.569108    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16699063-3cd1-4c81-87ec-a8d2497db12a" path="/var/lib/kubelet/pods/16699063-3cd1-4c81-87ec-a8d2497db12a/volumes"
Jul 27 19:58:21.795592 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:21.795471    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:24.229593 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.229560    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:24.334639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.334601    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"]
Jul 27 19:58:24.334979 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.334963    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf73465-b35c-4f0d-9299-7f786f4c6cd1" containerName="payload-pre-processing"
Jul 27 19:58:24.334979 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.334979    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf73465-b35c-4f0d-9299-7f786f4c6cd1" containerName="payload-pre-processing"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.334990    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16699063-3cd1-4c81-87ec-a8d2497db12a" containerName="revoke-keys"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.334998    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="16699063-3cd1-4c81-87ec-a8d2497db12a" containerName="revoke-keys"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335011    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b90b70d-9794-4174-ba29-32521c25b329" containerName="maas-api"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335018    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b90b70d-9794-4174-ba29-32521c25b329" containerName="maas-api"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335026    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13d8a42-a847-425b-90de-fce05452d40e" containerName="payload-processing"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335032    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13d8a42-a847-425b-90de-fce05452d40e" containerName="payload-processing"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335083    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13d8a42-a847-425b-90de-fce05452d40e" containerName="payload-processing"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335091    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b90b70d-9794-4174-ba29-32521c25b329" containerName="maas-api"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335098    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf73465-b35c-4f0d-9299-7f786f4c6cd1" containerName="payload-pre-processing"
Jul 27 19:58:24.335124 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.335104    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="16699063-3cd1-4c81-87ec-a8d2497db12a" containerName="revoke-keys"
Jul 27 19:58:24.339671 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.339651    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.342797 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.342777    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4369fd46-serving-cert\""
Jul 27 19:58:24.350452 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.350430    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"]
Jul 27 19:58:24.368885 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.368857    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:24.497639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.497519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-openshift-service-ca\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.497639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.497573    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4zt\" (UniqueName: \"kubernetes.io/projected/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-kube-api-access-jz4zt\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.497639 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.497591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.598783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.598746    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-openshift-service-ca\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.598783 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.598785    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4zt\" (UniqueName: \"kubernetes.io/projected/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-kube-api-access-jz4zt\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.598995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.598813    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.598995 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:58:24.598932    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-serving-cert: secret "maas-api-e2e-migrate-4369fd46-serving-cert" not found
Jul 27 19:58:24.599097 ip-10-0-143-3 kubenswrapper[2568]: E0727 19:58:24.599071    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls podName:5d6b86a9-9b95-4d70-8cde-9ad315ce3e11 nodeName:}" failed. No retries permitted until 2026-07-27 19:58:25.099006316 +0000 UTC m=+2054.094854290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls") pod "maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" (UID: "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11") : secret "maas-api-e2e-migrate-4369fd46-serving-cert" not found
Jul 27 19:58:24.599472 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.599455    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-openshift-service-ca\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.608550 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.608500    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4zt\" (UniqueName: \"kubernetes.io/projected/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-kube-api-access-jz4zt\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:24.744623 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.744590    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:24.915669 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:24.915635    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:25.026375 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.026343    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:25.103977 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.103937    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:25.106568 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.106546    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls\") pod \"maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:25.129921 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.129889    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"]
Jul 27 19:58:25.133570 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.133555    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.136436 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.136415    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4369fd46\""
Jul 27 19:58:25.136584 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.136468    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4369fd46-dockercfg-9zxn6\""
Jul 27 19:58:25.143875 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.143834    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"]
Jul 27 19:58:25.205452 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.205369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.205452 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.205420    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvzm\" (UniqueName: \"kubernetes.io/projected/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-kube-api-access-lrvzm\") pod \"payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.214494 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.214460    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"]
Jul 27 19:58:25.217836 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.217817    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.230200 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.230174    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"]
Jul 27 19:58:25.250139 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.250114    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:25.306874 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.306843    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvzm\" (UniqueName: \"kubernetes.io/projected/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-kube-api-access-lrvzm\") pod \"payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.307035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.306938    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp6nm\" (UniqueName: \"kubernetes.io/projected/8f677939-cd68-48df-bad2-af61871ba966-kube-api-access-mp6nm\") pod \"payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") " pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.307035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.306970    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f677939-cd68-48df-bad2-af61871ba966-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") " pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.307035 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.307010    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.307893 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.307829    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.318236 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.318207    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvzm\" (UniqueName: \"kubernetes.io/projected/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-kube-api-access-lrvzm\") pod \"payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.388809 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.388789    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"]
Jul 27 19:58:25.391001 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:58:25.390978    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6b86a9_9b95_4d70_8cde_9ad315ce3e11.slice/crio-e68a4a27b904381d3b778f11c1935b17831cc271a0a703d76264e6d39196be5b WatchSource:0}: Error finding container e68a4a27b904381d3b778f11c1935b17831cc271a0a703d76264e6d39196be5b: Status 404 returned error can't find the container with id e68a4a27b904381d3b778f11c1935b17831cc271a0a703d76264e6d39196be5b
Jul 27 19:58:25.407951 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.407922    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp6nm\" (UniqueName: \"kubernetes.io/projected/8f677939-cd68-48df-bad2-af61871ba966-kube-api-access-mp6nm\") pod \"payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") " pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.408050 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.407960    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f677939-cd68-48df-bad2-af61871ba966-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") " pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.408708 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.408686    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f677939-cd68-48df-bad2-af61871ba966-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") " pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.418338 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.418314    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp6nm\" (UniqueName: \"kubernetes.io/projected/8f677939-cd68-48df-bad2-af61871ba966-kube-api-access-mp6nm\") pod \"payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") " pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.444714 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.444689    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.527779 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.527697    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:25.583468 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.583346    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"]
Jul 27 19:58:25.586668 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:58:25.586642    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab686f0_4eba_4388_b5f1_ed934e9a5e5e.slice/crio-bb448deaa3e79297f6af5da8acd5d07262128b9c582a32adb42fda780d42649d WatchSource:0}: Error finding container bb448deaa3e79297f6af5da8acd5d07262128b9c582a32adb42fda780d42649d: Status 404 returned error can't find the container with id bb448deaa3e79297f6af5da8acd5d07262128b9c582a32adb42fda780d42649d
Jul 27 19:58:25.664867 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.664836    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" event={"ID":"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11","Type":"ContainerStarted","Data":"e68a4a27b904381d3b778f11c1935b17831cc271a0a703d76264e6d39196be5b"}
Jul 27 19:58:25.666200 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.666177    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n" event={"ID":"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e","Type":"ContainerStarted","Data":"fb01753397842f74e9c9242043d339e3437a66fb41f8ac1356830be7d4383ec5"}
Jul 27 19:58:25.666317 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.666223    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n" event={"ID":"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e","Type":"ContainerStarted","Data":"bb448deaa3e79297f6af5da8acd5d07262128b9c582a32adb42fda780d42649d"}
Jul 27 19:58:25.666766 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.666440    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:25.668995 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.668978    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"]
Jul 27 19:58:25.671586 ip-10-0-143-3 kubenswrapper[2568]: W0727 19:58:25.671560    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f677939_cd68_48df_bad2_af61871ba966.slice/crio-10f362bbd6669227ddc0e0a12564d0014273b6676e032972ddd334019a8cecc1 WatchSource:0}: Error finding container 10f362bbd6669227ddc0e0a12564d0014273b6676e032972ddd334019a8cecc1: Status 404 returned error can't find the container with id 10f362bbd6669227ddc0e0a12564d0014273b6676e032972ddd334019a8cecc1
Jul 27 19:58:25.685003 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:25.684945    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n" podStartSLOduration=0.684925754 podStartE2EDuration="684.925754ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:25.684377386 +0000 UTC m=+2054.680225390" watchObservedRunningTime="2026-07-27 19:58:25.684925754 +0000 UTC m=+2054.680773733"
Jul 27 19:58:26.673222 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:26.673184    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" event={"ID":"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11","Type":"ContainerStarted","Data":"a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00"}
Jul 27 19:58:26.673746 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:26.673407    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:26.675077 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:26.675047    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72" event={"ID":"8f677939-cd68-48df-bad2-af61871ba966","Type":"ContainerStarted","Data":"e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9"}
Jul 27 19:58:26.675230 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:26.675100    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:26.675230 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:26.675122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72" event={"ID":"8f677939-cd68-48df-bad2-af61871ba966","Type":"ContainerStarted","Data":"10f362bbd6669227ddc0e0a12564d0014273b6676e032972ddd334019a8cecc1"}
Jul 27 19:58:26.708428 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:26.708375    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" podStartSLOduration=2.100433356 podStartE2EDuration="2.708361177s" totalImagesPullingTime="607.927821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:26.706946992 +0000 UTC m=+2055.702794989" watchObservedRunningTime="2026-07-27 19:58:26.708361177 +0000 UTC m=+2055.704209194"
Jul 27 19:58:32.691179 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:32.691101    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 19:58:32.711227 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:32.711176    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72" podStartSLOduration=7.711161359 podStartE2EDuration="7.711161359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:26.739920495 +0000 UTC m=+2055.735768469" watchObservedRunningTime="2026-07-27 19:58:32.711161359 +0000 UTC m=+2061.707009336"
Jul 27 19:58:36.677712 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:36.677677    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 19:58:37.683037 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:37.683005    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 19:58:40.087487 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:40.087455    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:53.662262 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:53.662227    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:58:53.764057 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:58:53.764025    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:59:11.554675 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:59:11.554647    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:59:11.556158 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:59:11.556137    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 19:59:13.368541 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:59:13.368486    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 19:59:13.468629 ip-10-0-143-3 kubenswrapper[2568]: I0727 19:59:13.468597    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:00:00.144592 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.144547    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"]
Jul 27 20:00:00.148097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.148074    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"
Jul 27 20:00:00.149970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.149947    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"]
Jul 27 20:00:00.151027 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.151007    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:00:00.153165 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.153149    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"
Jul 27 20:00:00.158241 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.158217    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"]
Jul 27 20:00:00.159915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.159895    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"
Jul 27 20:00:00.161030 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.160998    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"]
Jul 27 20:00:00.168965 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.168945    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"
Jul 27 20:00:00.307810 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.307782    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"]
Jul 27 20:00:00.310651 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:00:00.310609    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0563b4_b881_4f3d_8cf3_5caa156b3f1d.slice/crio-ca80ec0fbd4b3e25dd720518acd82a356bf0209bc780d384839936428ba5a7ce WatchSource:0}: Error finding container ca80ec0fbd4b3e25dd720518acd82a356bf0209bc780d384839936428ba5a7ce: Status 404 returned error can't find the container with id ca80ec0fbd4b3e25dd720518acd82a356bf0209bc780d384839936428ba5a7ce
Jul 27 20:00:00.337071 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:00.337047    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"]
Jul 27 20:00:00.346587 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:00:00.346564    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea61ffc_a6ab_4553_a5ce_98c939a6840d.slice/crio-483a6839b1fa7d8698b52f26b83a69d5fcfe3d49d45094e6aeda8e014a0c10fa WatchSource:0}: Error finding container 483a6839b1fa7d8698b52f26b83a69d5fcfe3d49d45094e6aeda8e014a0c10fa: Status 404 returned error can't find the container with id 483a6839b1fa7d8698b52f26b83a69d5fcfe3d49d45094e6aeda8e014a0c10fa
Jul 27 20:00:01.108895 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:01.108862    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4" event={"ID":"8a0563b4-b881-4f3d-8cf3-5caa156b3f1d","Type":"ContainerStarted","Data":"8f7d7228684bc21dc5db888ec815a3860d626287f29dba0b88ced052303fb180"}
Jul 27 20:00:01.109068 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:01.108926    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4" event={"ID":"8a0563b4-b881-4f3d-8cf3-5caa156b3f1d","Type":"ContainerStarted","Data":"ca80ec0fbd4b3e25dd720518acd82a356bf0209bc780d384839936428ba5a7ce"}
Jul 27 20:00:01.110298 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:01.110274    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr" event={"ID":"2ea61ffc-a6ab-4553-a5ce-98c939a6840d","Type":"ContainerStarted","Data":"707e54e97c584dc8cc17bd4aa8a0575397ad1894664f44196b662576ee537464"}
Jul 27 20:00:01.110408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:01.110316    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr" event={"ID":"2ea61ffc-a6ab-4553-a5ce-98c939a6840d","Type":"ContainerStarted","Data":"483a6839b1fa7d8698b52f26b83a69d5fcfe3d49d45094e6aeda8e014a0c10fa"}
Jul 27 20:00:01.126242 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:01.126138    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4" podStartSLOduration=1.126119622 podStartE2EDuration="1.126119622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:00:01.125344292 +0000 UTC m=+2150.121192281" watchObservedRunningTime="2026-07-27 20:00:01.126119622 +0000 UTC m=+2150.121967599"
Jul 27 20:00:01.142126 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:01.142081    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr" podStartSLOduration=1.142067036 podStartE2EDuration="1.142067036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:00:01.141514022 +0000 UTC m=+2150.137362024" watchObservedRunningTime="2026-07-27 20:00:01.142067036 +0000 UTC m=+2150.137915013"
Jul 27 20:00:02.116808 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:02.116733    2568 generic.go:356] "Generic (PLEG): container finished" podID="8a0563b4-b881-4f3d-8cf3-5caa156b3f1d" containerID="8f7d7228684bc21dc5db888ec815a3860d626287f29dba0b88ced052303fb180" exitCode=0
Jul 27 20:00:02.117246 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:02.116827    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4" event={"ID":"8a0563b4-b881-4f3d-8cf3-5caa156b3f1d","Type":"ContainerDied","Data":"8f7d7228684bc21dc5db888ec815a3860d626287f29dba0b88ced052303fb180"}
Jul 27 20:00:02.118297 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:02.118274    2568 generic.go:356] "Generic (PLEG): container finished" podID="2ea61ffc-a6ab-4553-a5ce-98c939a6840d" containerID="707e54e97c584dc8cc17bd4aa8a0575397ad1894664f44196b662576ee537464" exitCode=0
Jul 27 20:00:02.118398 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:02.118301    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr" event={"ID":"2ea61ffc-a6ab-4553-a5ce-98c939a6840d","Type":"ContainerDied","Data":"707e54e97c584dc8cc17bd4aa8a0575397ad1894664f44196b662576ee537464"}
Jul 27 20:00:03.265947 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:03.265914    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"
Jul 27 20:00:03.285720 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:03.285691    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"
Jul 27 20:00:04.129153 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:04.129121    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"
Jul 27 20:00:04.129153 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:04.129135    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4" event={"ID":"8a0563b4-b881-4f3d-8cf3-5caa156b3f1d","Type":"ContainerDied","Data":"ca80ec0fbd4b3e25dd720518acd82a356bf0209bc780d384839936428ba5a7ce"}
Jul 27 20:00:04.129400 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:04.129211    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca80ec0fbd4b3e25dd720518acd82a356bf0209bc780d384839936428ba5a7ce"
Jul 27 20:00:04.130416 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:04.130395    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr" event={"ID":"2ea61ffc-a6ab-4553-a5ce-98c939a6840d","Type":"ContainerDied","Data":"483a6839b1fa7d8698b52f26b83a69d5fcfe3d49d45094e6aeda8e014a0c10fa"}
Jul 27 20:00:04.130593 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:04.130419    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-ct4kr"
Jul 27 20:00:04.130593 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:00:04.130429    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483a6839b1fa7d8698b52f26b83a69d5fcfe3d49d45094e6aeda8e014a0c10fa"
Jul 27 20:01:20.057246 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:20.057208    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:32.806543 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.806430    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"]
Jul 27 20:01:32.807021 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.807000    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea61ffc-a6ab-4553-a5ce-98c939a6840d" containerName="cleanup"
Jul 27 20:01:32.807098 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.807025    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea61ffc-a6ab-4553-a5ce-98c939a6840d" containerName="cleanup"
Jul 27 20:01:32.807098 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.807043    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0563b4-b881-4f3d-8cf3-5caa156b3f1d" containerName="cleanup"
Jul 27 20:01:32.807098 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.807052    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0563b4-b881-4f3d-8cf3-5caa156b3f1d" containerName="cleanup"
Jul 27 20:01:32.807253 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.807179    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0563b4-b881-4f3d-8cf3-5caa156b3f1d" containerName="cleanup"
Jul 27 20:01:32.807253 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.807193    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea61ffc-a6ab-4553-a5ce-98c939a6840d" containerName="cleanup"
Jul 27 20:01:32.810242 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.810220    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:32.813047 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.813027    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:01:32.818270 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.817785    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"]
Jul 27 20:01:32.890616 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.890578    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d20a53-a37d-47c0-89af-1eb771a7aa08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf\" (UID: \"00d20a53-a37d-47c0-89af-1eb771a7aa08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:32.991557 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.991483    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d20a53-a37d-47c0-89af-1eb771a7aa08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf\" (UID: \"00d20a53-a37d-47c0-89af-1eb771a7aa08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:32.992158 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:32.992135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d20a53-a37d-47c0-89af-1eb771a7aa08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf\" (UID: \"00d20a53-a37d-47c0-89af-1eb771a7aa08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:33.122901 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:33.122811    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:33.262673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:33.262645    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"]
Jul 27 20:01:33.265353 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:01:33.265326    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d20a53_a37d_47c0_89af_1eb771a7aa08.slice/crio-10e97df01d77721e260b96d0da72ef28a7dc3ce8a1e95d05ea5f93edcbe7758d WatchSource:0}: Error finding container 10e97df01d77721e260b96d0da72ef28a7dc3ce8a1e95d05ea5f93edcbe7758d: Status 404 returned error can't find the container with id 10e97df01d77721e260b96d0da72ef28a7dc3ce8a1e95d05ea5f93edcbe7758d
Jul 27 20:01:33.556976 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:33.556936    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf" event={"ID":"00d20a53-a37d-47c0-89af-1eb771a7aa08","Type":"ContainerStarted","Data":"10e97df01d77721e260b96d0da72ef28a7dc3ce8a1e95d05ea5f93edcbe7758d"}
Jul 27 20:01:34.563591 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:34.563558    2568 generic.go:356] "Generic (PLEG): container finished" podID="00d20a53-a37d-47c0-89af-1eb771a7aa08" containerID="23578552d88bd310a76fb8abdea0e6384092f6e4a654e04eb7acfd92f962439d" exitCode=0
Jul 27 20:01:34.563988 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:34.563638    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf" event={"ID":"00d20a53-a37d-47c0-89af-1eb771a7aa08","Type":"ContainerDied","Data":"23578552d88bd310a76fb8abdea0e6384092f6e4a654e04eb7acfd92f962439d"}
Jul 27 20:01:35.702340 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:35.702315    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:35.817366 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:35.817324    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d20a53-a37d-47c0-89af-1eb771a7aa08-maas-api-service-ca\") pod \"00d20a53-a37d-47c0-89af-1eb771a7aa08\" (UID: \"00d20a53-a37d-47c0-89af-1eb771a7aa08\") "
Jul 27 20:01:35.817759 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:35.817736    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d20a53-a37d-47c0-89af-1eb771a7aa08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00d20a53-a37d-47c0-89af-1eb771a7aa08" (UID: "00d20a53-a37d-47c0-89af-1eb771a7aa08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:01:35.918972 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:35.918886    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d20a53-a37d-47c0-89af-1eb771a7aa08-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:36.577723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:36.577678    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf" event={"ID":"00d20a53-a37d-47c0-89af-1eb771a7aa08","Type":"ContainerDied","Data":"10e97df01d77721e260b96d0da72ef28a7dc3ce8a1e95d05ea5f93edcbe7758d"}
Jul 27 20:01:36.577930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:36.577782    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e97df01d77721e260b96d0da72ef28a7dc3ce8a1e95d05ea5f93edcbe7758d"
Jul 27 20:01:36.577930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:36.577863    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"
Jul 27 20:01:43.280073 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.280040    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"]
Jul 27 20:01:43.280492 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.280232    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" podUID="5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" containerName="maas-api" containerID="cri-o://a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00" gracePeriod=30
Jul 27 20:01:43.304882 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.304847    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:43.305722 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.305700    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"]
Jul 27 20:01:43.309373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.309353    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-4369fd46-29753040-6btm4"]
Jul 27 20:01:43.336665 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.336630    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"]
Jul 27 20:01:43.336965 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.336914    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72" podUID="8f677939-cd68-48df-bad2-af61871ba966" containerName="payload-processing" containerID="cri-o://e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9" gracePeriod=30
Jul 27 20:01:43.344370 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.344067    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"]
Jul 27 20:01:43.344370 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.344305    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n" podUID="7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" containerName="payload-pre-processing" containerID="cri-o://fb01753397842f74e9c9242043d339e3437a66fb41f8ac1356830be7d4383ec5" gracePeriod=30
Jul 27 20:01:43.378938 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.378890    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:43.444680 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.444650    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:43.571509 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.571470    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0563b4-b881-4f3d-8cf3-5caa156b3f1d" path="/var/lib/kubelet/pods/8a0563b4-b881-4f3d-8cf3-5caa156b3f1d/volumes"
Jul 27 20:01:43.602118 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.602090    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 20:01:43.605154 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.605128    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 20:01:43.616903 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.616876    2568 generic.go:356] "Generic (PLEG): container finished" podID="5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" containerID="a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00" exitCode=0
Jul 27 20:01:43.617075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.616921    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"
Jul 27 20:01:43.617075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.616960    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" event={"ID":"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11","Type":"ContainerDied","Data":"a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00"}
Jul 27 20:01:43.617075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.617017    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np" event={"ID":"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11","Type":"ContainerDied","Data":"e68a4a27b904381d3b778f11c1935b17831cc271a0a703d76264e6d39196be5b"}
Jul 27 20:01:43.617075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.617050    2568 scope.go:117] "RemoveContainer" containerID="a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00"
Jul 27 20:01:43.618365 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.618338    2568 generic.go:356] "Generic (PLEG): container finished" podID="8f677939-cd68-48df-bad2-af61871ba966" containerID="e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9" exitCode=0
Jul 27 20:01:43.618489 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.618389    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"
Jul 27 20:01:43.618489 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.618433    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72" event={"ID":"8f677939-cd68-48df-bad2-af61871ba966","Type":"ContainerDied","Data":"e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9"}
Jul 27 20:01:43.618489 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.618485    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72" event={"ID":"8f677939-cd68-48df-bad2-af61871ba966","Type":"ContainerDied","Data":"10f362bbd6669227ddc0e0a12564d0014273b6676e032972ddd334019a8cecc1"}
Jul 27 20:01:43.620331 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.620308    2568 generic.go:356] "Generic (PLEG): container finished" podID="7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" containerID="fb01753397842f74e9c9242043d339e3437a66fb41f8ac1356830be7d4383ec5" exitCode=0
Jul 27 20:01:43.620457 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.620398    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n" event={"ID":"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e","Type":"ContainerDied","Data":"fb01753397842f74e9c9242043d339e3437a66fb41f8ac1356830be7d4383ec5"}
Jul 27 20:01:43.628073 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.628048    2568 scope.go:117] "RemoveContainer" containerID="a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00"
Jul 27 20:01:43.628397 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:01:43.628355    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00\": container with ID starting with a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00 not found: ID does not exist" containerID="a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00"
Jul 27 20:01:43.628397 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.628386    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00"} err="failed to get container status \"a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00\": rpc error: code = NotFound desc = could not find container \"a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00\": container with ID starting with a3eef151398b8b1a4a9b62cc016d70a879ba99805598c44031e7fe9121ea0b00 not found: ID does not exist"
Jul 27 20:01:43.628590 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.628409    2568 scope.go:117] "RemoveContainer" containerID="e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9"
Jul 27 20:01:43.638821 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.638803    2568 scope.go:117] "RemoveContainer" containerID="e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9"
Jul 27 20:01:43.639113 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:01:43.639095    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9\": container with ID starting with e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9 not found: ID does not exist" containerID="e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9"
Jul 27 20:01:43.639184 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.639133    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9"} err="failed to get container status \"e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9\": rpc error: code = NotFound desc = could not find container \"e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9\": container with ID starting with e615e70642e10af79905718448b7c9cbf5b86765c1a504466dbb1333554af9b9 not found: ID does not exist"
Jul 27 20:01:43.649009 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.648985    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 20:01:43.688217 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688184    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f677939-cd68-48df-bad2-af61871ba966-plugins-config-volume\") pod \"8f677939-cd68-48df-bad2-af61871ba966\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") "
Jul 27 20:01:43.688379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688237    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls\") pod \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") "
Jul 27 20:01:43.688379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688275    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz4zt\" (UniqueName: \"kubernetes.io/projected/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-kube-api-access-jz4zt\") pod \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") "
Jul 27 20:01:43.688379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688308    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-openshift-service-ca\") pod \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\" (UID: \"5d6b86a9-9b95-4d70-8cde-9ad315ce3e11\") "
Jul 27 20:01:43.688379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688366    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp6nm\" (UniqueName: \"kubernetes.io/projected/8f677939-cd68-48df-bad2-af61871ba966-kube-api-access-mp6nm\") pod \"8f677939-cd68-48df-bad2-af61871ba966\" (UID: \"8f677939-cd68-48df-bad2-af61871ba966\") "
Jul 27 20:01:43.688688 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688661    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f677939-cd68-48df-bad2-af61871ba966-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f677939-cd68-48df-bad2-af61871ba966" (UID: "8f677939-cd68-48df-bad2-af61871ba966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:01:43.688816 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.688791    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" (UID: "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:01:43.690700 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.690678    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f677939-cd68-48df-bad2-af61871ba966-kube-api-access-mp6nm" (OuterVolumeSpecName: "kube-api-access-mp6nm") pod "8f677939-cd68-48df-bad2-af61871ba966" (UID: "8f677939-cd68-48df-bad2-af61871ba966"). InnerVolumeSpecName "kube-api-access-mp6nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:01:43.690789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.690683    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" (UID: "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:01:43.690789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.690699    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-kube-api-access-jz4zt" (OuterVolumeSpecName: "kube-api-access-jz4zt") pod "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" (UID: "5d6b86a9-9b95-4d70-8cde-9ad315ce3e11"). InnerVolumeSpecName "kube-api-access-jz4zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:01:43.789275 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789243    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-plugins-config-volume\") pod \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") "
Jul 27 20:01:43.789435 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789312    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrvzm\" (UniqueName: \"kubernetes.io/projected/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-kube-api-access-lrvzm\") pod \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\" (UID: \"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e\") "
Jul 27 20:01:43.789514 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789503    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp6nm\" (UniqueName: \"kubernetes.io/projected/8f677939-cd68-48df-bad2-af61871ba966-kube-api-access-mp6nm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.789588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789516    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f677939-cd68-48df-bad2-af61871ba966-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.789588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789549    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.789588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789559    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4zt\" (UniqueName: \"kubernetes.io/projected/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-kube-api-access-jz4zt\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.789588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789567    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.789724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.789624    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" (UID: "7ab686f0-4eba-4388-b5f1-ed934e9a5e5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:01:43.791417 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.791399    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-kube-api-access-lrvzm" (OuterVolumeSpecName: "kube-api-access-lrvzm") pod "7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" (UID: "7ab686f0-4eba-4388-b5f1-ed934e9a5e5e"). InnerVolumeSpecName "kube-api-access-lrvzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:01:43.890941 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.890851    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.890941 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.890880    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrvzm\" (UniqueName: \"kubernetes.io/projected/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e-kube-api-access-lrvzm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:01:43.943117 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.943083    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"]
Jul 27 20:01:43.947908 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.947883    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4369fd46-cfc9dd577-n6c72"]
Jul 27 20:01:43.960371 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.960350    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"]
Jul 27 20:01:43.965782 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:43.965759    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4369fd46-69d6457c9f-8n6np"]
Jul 27 20:01:44.628509 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:44.628428    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"
Jul 27 20:01:44.628509 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:44.628456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n" event={"ID":"7ab686f0-4eba-4388-b5f1-ed934e9a5e5e","Type":"ContainerDied","Data":"bb448deaa3e79297f6af5da8acd5d07262128b9c582a32adb42fda780d42649d"}
Jul 27 20:01:44.628509 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:44.628508    2568 scope.go:117] "RemoveContainer" containerID="fb01753397842f74e9c9242043d339e3437a66fb41f8ac1356830be7d4383ec5"
Jul 27 20:01:44.651793 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:44.651765    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"]
Jul 27 20:01:44.658464 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:44.658440    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4369fd46-777c944c9b-prf7n"]
Jul 27 20:01:45.567941 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:45.567908    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" path="/var/lib/kubelet/pods/5d6b86a9-9b95-4d70-8cde-9ad315ce3e11/volumes"
Jul 27 20:01:45.568277 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:45.568264    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" path="/var/lib/kubelet/pods/7ab686f0-4eba-4388-b5f1-ed934e9a5e5e/volumes"
Jul 27 20:01:45.568594 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:45.568582    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f677939-cd68-48df-bad2-af61871ba966" path="/var/lib/kubelet/pods/8f677939-cd68-48df-bad2-af61871ba966/volumes"
Jul 27 20:01:47.895148 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:47.895111    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:47.911330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:47.911289    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"]
Jul 27 20:01:47.917233 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:47.917207    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4369fd46-59hqf"]
Jul 27 20:01:48.059630 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:48.059602    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:48.278047 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:48.278010    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:48.505396 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:48.505368    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:48.829297 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:48.829239    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:48.953933 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:48.953897    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:49.567660 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:49.567624    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d20a53-a37d-47c0-89af-1eb771a7aa08" path="/var/lib/kubelet/pods/00d20a53-a37d-47c0-89af-1eb771a7aa08/volumes"
Jul 27 20:01:51.178444 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.178416    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:51.257313 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257275    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"]
Jul 27 20:01:51.257881 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257846    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" containerName="payload-pre-processing"
Jul 27 20:01:51.257881 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257873    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" containerName="payload-pre-processing"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257897    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f677939-cd68-48df-bad2-af61871ba966" containerName="payload-processing"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257906    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f677939-cd68-48df-bad2-af61871ba966" containerName="payload-processing"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257916    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d20a53-a37d-47c0-89af-1eb771a7aa08" containerName="revoke-keys"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257925    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d20a53-a37d-47c0-89af-1eb771a7aa08" containerName="revoke-keys"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257936    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" containerName="maas-api"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.257944    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" containerName="maas-api"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.258035    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6b86a9-9b95-4d70-8cde-9ad315ce3e11" containerName="maas-api"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.258050    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab686f0-4eba-4388-b5f1-ed934e9a5e5e" containerName="payload-pre-processing"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.258063    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f677939-cd68-48df-bad2-af61871ba966" containerName="payload-processing"
Jul 27 20:01:51.258075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.258075    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d20a53-a37d-47c0-89af-1eb771a7aa08" containerName="revoke-keys"
Jul 27 20:01:51.258501 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.258398    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:51.262668 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.262644    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.265387 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.265366    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-554d51aa-serving-cert\""
Jul 27 20:01:51.273958 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.273938    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"]
Jul 27 20:01:51.325364 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.325333    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:51.361169 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.361145    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.361292 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.361178    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322d2b78-70ae-408e-b76f-5a2d3b399319-openshift-service-ca\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.361292 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.361216    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jxj\" (UniqueName: \"kubernetes.io/projected/322d2b78-70ae-408e-b76f-5a2d3b399319-kube-api-access-z6jxj\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.462618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.462543    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.462618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.462587    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322d2b78-70ae-408e-b76f-5a2d3b399319-openshift-service-ca\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.462814 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:01:51.462630    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-serving-cert: secret "maas-api-e2e-ait-554d51aa-serving-cert" not found
Jul 27 20:01:51.462814 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.462631    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jxj\" (UniqueName: \"kubernetes.io/projected/322d2b78-70ae-408e-b76f-5a2d3b399319-kube-api-access-z6jxj\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.462814 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:01:51.462689    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls podName:322d2b78-70ae-408e-b76f-5a2d3b399319 nodeName:}" failed. No retries permitted until 2026-07-27 20:01:51.962672207 +0000 UTC m=+2260.958520163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls") pod "maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" (UID: "322d2b78-70ae-408e-b76f-5a2d3b399319") : secret "maas-api-e2e-ait-554d51aa-serving-cert" not found
Jul 27 20:01:51.463238 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.463220    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322d2b78-70ae-408e-b76f-5a2d3b399319-openshift-service-ca\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.475424 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.475393    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jxj\" (UniqueName: \"kubernetes.io/projected/322d2b78-70ae-408e-b76f-5a2d3b399319-kube-api-access-z6jxj\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.846337 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.846305    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:51.942883 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.942846    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:51.966729 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.966702    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.969274 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.969254    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls\") pod \"maas-api-e2e-ait-554d51aa-588f7887f8-vb46c\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:51.994735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.994696    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"]
Jul 27 20:01:51.998730 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:51.998709    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.001448 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.001424    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-554d51aa\""
Jul 27 20:01:52.001587 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.001449    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-554d51aa-dockercfg-94hgf\""
Jul 27 20:01:52.006833 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.006814    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"]
Jul 27 20:01:52.010583 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.010562    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.010583 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.010573    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"]
Jul 27 20:01:52.020539 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.019670    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"]
Jul 27 20:01:52.046804 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.046780    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:01:52.168408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.168323    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwbf\" (UniqueName: \"kubernetes.io/projected/58f6533a-0bbc-4899-9556-1fb69d45137f-kube-api-access-gjwbf\") pod \"payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") " pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.168408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.168373    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7410d9f7-2e12-4514-a168-5485f7c76558-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.168408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.168399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwkz7\" (UniqueName: \"kubernetes.io/projected/7410d9f7-2e12-4514-a168-5485f7c76558-kube-api-access-dwkz7\") pod \"payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.168684 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.168468    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f6533a-0bbc-4899-9556-1fb69d45137f-plugins-config-volume\") pod \"payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") " pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.174204 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.174189    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:01:52.269253 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.269217    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f6533a-0bbc-4899-9556-1fb69d45137f-plugins-config-volume\") pod \"payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") " pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.269719 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.269337    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwbf\" (UniqueName: \"kubernetes.io/projected/58f6533a-0bbc-4899-9556-1fb69d45137f-kube-api-access-gjwbf\") pod \"payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") " pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.269719 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.269388    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7410d9f7-2e12-4514-a168-5485f7c76558-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.269719 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.269426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwkz7\" (UniqueName: \"kubernetes.io/projected/7410d9f7-2e12-4514-a168-5485f7c76558-kube-api-access-dwkz7\") pod \"payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.270096 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.270075    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f6533a-0bbc-4899-9556-1fb69d45137f-plugins-config-volume\") pod \"payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") " pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.270379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.270356    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7410d9f7-2e12-4514-a168-5485f7c76558-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.278999 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.278972    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwkz7\" (UniqueName: \"kubernetes.io/projected/7410d9f7-2e12-4514-a168-5485f7c76558-kube-api-access-dwkz7\") pod \"payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.279136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.279076    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwbf\" (UniqueName: \"kubernetes.io/projected/58f6533a-0bbc-4899-9556-1fb69d45137f-kube-api-access-gjwbf\") pod \"payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") " pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.311237 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.311214    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"]
Jul 27 20:01:52.312372 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.312353    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.314084 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:01:52.314060    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322d2b78_70ae_408e_b76f_5a2d3b399319.slice/crio-8a75a27413cca98906d581e4a0b94134b6a107ac5e8fd771bb3a7ddd00edac02 WatchSource:0}: Error finding container 8a75a27413cca98906d581e4a0b94134b6a107ac5e8fd771bb3a7ddd00edac02: Status 404 returned error can't find the container with id 8a75a27413cca98906d581e4a0b94134b6a107ac5e8fd771bb3a7ddd00edac02
Jul 27 20:01:52.324731 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.324708    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.466990 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.466915    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"]
Jul 27 20:01:52.468985 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:01:52.468957    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7410d9f7_2e12_4514_a168_5485f7c76558.slice/crio-f2e95c1493754213835c61c2848c5babe18f5e5407838d2cdc40bfcfcc625a50 WatchSource:0}: Error finding container f2e95c1493754213835c61c2848c5babe18f5e5407838d2cdc40bfcfcc625a50: Status 404 returned error can't find the container with id f2e95c1493754213835c61c2848c5babe18f5e5407838d2cdc40bfcfcc625a50
Jul 27 20:01:52.490257 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.490230    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"]
Jul 27 20:01:52.493240 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:01:52.493211    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f6533a_0bbc_4899_9556_1fb69d45137f.slice/crio-d5a1c48eaeb8855e07a956c0e35df737b3b58ed081a7484872095b68b3e361d0 WatchSource:0}: Error finding container d5a1c48eaeb8855e07a956c0e35df737b3b58ed081a7484872095b68b3e361d0: Status 404 returned error can't find the container with id d5a1c48eaeb8855e07a956c0e35df737b3b58ed081a7484872095b68b3e361d0
Jul 27 20:01:52.673398 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.673362    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8" event={"ID":"7410d9f7-2e12-4514-a168-5485f7c76558","Type":"ContainerStarted","Data":"d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4"}
Jul 27 20:01:52.673628 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.673429    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8" event={"ID":"7410d9f7-2e12-4514-a168-5485f7c76558","Type":"ContainerStarted","Data":"f2e95c1493754213835c61c2848c5babe18f5e5407838d2cdc40bfcfcc625a50"}
Jul 27 20:01:52.673705 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.673662    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:01:52.674839 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.674812    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" event={"ID":"322d2b78-70ae-408e-b76f-5a2d3b399319","Type":"ContainerStarted","Data":"8a75a27413cca98906d581e4a0b94134b6a107ac5e8fd771bb3a7ddd00edac02"}
Jul 27 20:01:52.676365 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.676338    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb" event={"ID":"58f6533a-0bbc-4899-9556-1fb69d45137f","Type":"ContainerStarted","Data":"5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8"}
Jul 27 20:01:52.676460 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.676387    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb" event={"ID":"58f6533a-0bbc-4899-9556-1fb69d45137f","Type":"ContainerStarted","Data":"d5a1c48eaeb8855e07a956c0e35df737b3b58ed081a7484872095b68b3e361d0"}
Jul 27 20:01:52.676460 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.676416    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:01:52.699229 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.699140    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8" podStartSLOduration=1.699126605 podStartE2EDuration="1.699126605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:01:52.698436664 +0000 UTC m=+2261.694284654" watchObservedRunningTime="2026-07-27 20:01:52.699126605 +0000 UTC m=+2261.694974583"
Jul 27 20:01:52.741583 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:52.741505    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb" podStartSLOduration=1.741489555 podStartE2EDuration="1.741489555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:01:52.741177748 +0000 UTC m=+2261.737025725" watchObservedRunningTime="2026-07-27 20:01:52.741489555 +0000 UTC m=+2261.737337533"
Jul 27 20:01:53.682766 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:53.682727    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" event={"ID":"322d2b78-70ae-408e-b76f-5a2d3b399319","Type":"ContainerStarted","Data":"007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1"}
Jul 27 20:01:53.705978 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:53.705932    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" podStartSLOduration=2.179247261 podStartE2EDuration="2.70590541s" totalImagesPullingTime="526.658149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:01:53.704789758 +0000 UTC m=+2262.700637739" watchObservedRunningTime="2026-07-27 20:01:53.70590541 +0000 UTC m=+2262.701753388"
Jul 27 20:01:54.687369 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:01:54.687334    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:02:00.697572 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:02:00.697541    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:02:03.685098 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:02:03.685069    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:02:03.685464 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:02:03.685142    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:04:11.585262 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:04:11.585229    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:04:11.587275 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:04:11.587253    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:04:11.684906 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:04:11.684860    2568 scope.go:117] "RemoveContainer" containerID="1ee3b18281bf1a433f956766ad5599aa158e2bcc104f75fd41718c8569bced88"
Jul 27 20:04:32.273729 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:04:32.273659    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:06:11.714622 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:11.714592    2568 scope.go:117] "RemoveContainer" containerID="8f7d7228684bc21dc5db888ec815a3860d626287f29dba0b88ced052303fb180"
Jul 27 20:06:24.506448 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:24.506419    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:06:35.377732 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:35.377700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:06:57.604100 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.604068    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"]
Jul 27 20:06:57.607568 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.607548    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:06:57.610576 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.610554    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:06:57.623732 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.623708    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"]
Jul 27 20:06:57.710781 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.710744    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d727ea9-cd93-4d2b-a01e-968b9fddcbad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-554d51aa-4t75j\" (UID: \"4d727ea9-cd93-4d2b-a01e-968b9fddcbad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:06:57.812213 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.812173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d727ea9-cd93-4d2b-a01e-968b9fddcbad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-554d51aa-4t75j\" (UID: \"4d727ea9-cd93-4d2b-a01e-968b9fddcbad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:06:57.812825 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.812805    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d727ea9-cd93-4d2b-a01e-968b9fddcbad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-554d51aa-4t75j\" (UID: \"4d727ea9-cd93-4d2b-a01e-968b9fddcbad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:06:57.917746 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:57.917649    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:06:58.074298 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:06:58.074269    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d727ea9_cd93_4d2b_a01e_968b9fddcbad.slice/crio-491c67ca1beff63c83664a0af9d792e1e31d47ea3405dc4fd52e5f4dcff5c1e0 WatchSource:0}: Error finding container 491c67ca1beff63c83664a0af9d792e1e31d47ea3405dc4fd52e5f4dcff5c1e0: Status 404 returned error can't find the container with id 491c67ca1beff63c83664a0af9d792e1e31d47ea3405dc4fd52e5f4dcff5c1e0
Jul 27 20:06:58.074630 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:58.074604    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"]
Jul 27 20:06:58.141162 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:58.141127    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j" event={"ID":"4d727ea9-cd93-4d2b-a01e-968b9fddcbad","Type":"ContainerStarted","Data":"491c67ca1beff63c83664a0af9d792e1e31d47ea3405dc4fd52e5f4dcff5c1e0"}
Jul 27 20:06:59.147976 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:59.147939    2568 generic.go:356] "Generic (PLEG): container finished" podID="4d727ea9-cd93-4d2b-a01e-968b9fddcbad" containerID="da9cd8b0c85b87f7078f4f0716e82767fce68930ab6ce4b5e7cb5194009f8c15" exitCode=0
Jul 27 20:06:59.148370 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:06:59.147982    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j" event={"ID":"4d727ea9-cd93-4d2b-a01e-968b9fddcbad","Type":"ContainerDied","Data":"da9cd8b0c85b87f7078f4f0716e82767fce68930ab6ce4b5e7cb5194009f8c15"}
Jul 27 20:07:00.281435 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:00.281413    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:07:00.332993 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:00.332964    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d727ea9-cd93-4d2b-a01e-968b9fddcbad-maas-api-service-ca\") pod \"4d727ea9-cd93-4d2b-a01e-968b9fddcbad\" (UID: \"4d727ea9-cd93-4d2b-a01e-968b9fddcbad\") "
Jul 27 20:07:00.333362 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:00.333337    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d727ea9-cd93-4d2b-a01e-968b9fddcbad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d727ea9-cd93-4d2b-a01e-968b9fddcbad" (UID: "4d727ea9-cd93-4d2b-a01e-968b9fddcbad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:00.434649 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:00.434561    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d727ea9-cd93-4d2b-a01e-968b9fddcbad-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:01.157972 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:01.157945    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"
Jul 27 20:07:01.158171 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:01.157991    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j" event={"ID":"4d727ea9-cd93-4d2b-a01e-968b9fddcbad","Type":"ContainerDied","Data":"491c67ca1beff63c83664a0af9d792e1e31d47ea3405dc4fd52e5f4dcff5c1e0"}
Jul 27 20:07:01.158171 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:01.158045    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491c67ca1beff63c83664a0af9d792e1e31d47ea3405dc4fd52e5f4dcff5c1e0"
Jul 27 20:07:08.095180 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.095146    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"]
Jul 27 20:07:08.095704 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.095370    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" podUID="322d2b78-70ae-408e-b76f-5a2d3b399319" containerName="maas-api" containerID="cri-o://007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1" gracePeriod=30
Jul 27 20:07:08.107389 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.107365    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:08.150061 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.150029    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"]
Jul 27 20:07:08.150244 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.150222    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb" podUID="58f6533a-0bbc-4899-9556-1fb69d45137f" containerName="payload-processing" containerID="cri-o://5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8" gracePeriod=30
Jul 27 20:07:08.160415 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.160389    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"]
Jul 27 20:07:08.160733 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.160689    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8" podUID="7410d9f7-2e12-4514-a168-5485f7c76558" containerName="payload-pre-processing" containerID="cri-o://d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4" gracePeriod=30
Jul 27 20:07:08.195034 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.194989    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:08.254134 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.254108    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:08.407621 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.407594    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:07:08.430361 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.430264    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:07:08.445708 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.445689    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:07:08.507105 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507069    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6jxj\" (UniqueName: \"kubernetes.io/projected/322d2b78-70ae-408e-b76f-5a2d3b399319-kube-api-access-z6jxj\") pod \"322d2b78-70ae-408e-b76f-5a2d3b399319\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") "
Jul 27 20:07:08.507281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507147    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322d2b78-70ae-408e-b76f-5a2d3b399319-openshift-service-ca\") pod \"322d2b78-70ae-408e-b76f-5a2d3b399319\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") "
Jul 27 20:07:08.507281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507176    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls\") pod \"322d2b78-70ae-408e-b76f-5a2d3b399319\" (UID: \"322d2b78-70ae-408e-b76f-5a2d3b399319\") "
Jul 27 20:07:08.507281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507197    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjwbf\" (UniqueName: \"kubernetes.io/projected/58f6533a-0bbc-4899-9556-1fb69d45137f-kube-api-access-gjwbf\") pod \"58f6533a-0bbc-4899-9556-1fb69d45137f\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") "
Jul 27 20:07:08.507281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507223    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwkz7\" (UniqueName: \"kubernetes.io/projected/7410d9f7-2e12-4514-a168-5485f7c76558-kube-api-access-dwkz7\") pod \"7410d9f7-2e12-4514-a168-5485f7c76558\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") "
Jul 27 20:07:08.507281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507254    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7410d9f7-2e12-4514-a168-5485f7c76558-plugins-config-volume\") pod \"7410d9f7-2e12-4514-a168-5485f7c76558\" (UID: \"7410d9f7-2e12-4514-a168-5485f7c76558\") "
Jul 27 20:07:08.507598 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507318    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f6533a-0bbc-4899-9556-1fb69d45137f-plugins-config-volume\") pod \"58f6533a-0bbc-4899-9556-1fb69d45137f\" (UID: \"58f6533a-0bbc-4899-9556-1fb69d45137f\") "
Jul 27 20:07:08.507673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507591    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322d2b78-70ae-408e-b76f-5a2d3b399319-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "322d2b78-70ae-408e-b76f-5a2d3b399319" (UID: "322d2b78-70ae-408e-b76f-5a2d3b399319"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:08.507947 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507699    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7410d9f7-2e12-4514-a168-5485f7c76558-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7410d9f7-2e12-4514-a168-5485f7c76558" (UID: "7410d9f7-2e12-4514-a168-5485f7c76558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:08.507947 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.507916    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f6533a-0bbc-4899-9556-1fb69d45137f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58f6533a-0bbc-4899-9556-1fb69d45137f" (UID: "58f6533a-0bbc-4899-9556-1fb69d45137f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:07:08.509395 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.509367    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "322d2b78-70ae-408e-b76f-5a2d3b399319" (UID: "322d2b78-70ae-408e-b76f-5a2d3b399319"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:07:08.509621 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.509441    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f6533a-0bbc-4899-9556-1fb69d45137f-kube-api-access-gjwbf" (OuterVolumeSpecName: "kube-api-access-gjwbf") pod "58f6533a-0bbc-4899-9556-1fb69d45137f" (UID: "58f6533a-0bbc-4899-9556-1fb69d45137f"). InnerVolumeSpecName "kube-api-access-gjwbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:08.509825 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.509804    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322d2b78-70ae-408e-b76f-5a2d3b399319-kube-api-access-z6jxj" (OuterVolumeSpecName: "kube-api-access-z6jxj") pod "322d2b78-70ae-408e-b76f-5a2d3b399319" (UID: "322d2b78-70ae-408e-b76f-5a2d3b399319"). InnerVolumeSpecName "kube-api-access-z6jxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:08.509825 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.509809    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7410d9f7-2e12-4514-a168-5485f7c76558-kube-api-access-dwkz7" (OuterVolumeSpecName: "kube-api-access-dwkz7") pod "7410d9f7-2e12-4514-a168-5485f7c76558" (UID: "7410d9f7-2e12-4514-a168-5485f7c76558"). InnerVolumeSpecName "kube-api-access-dwkz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608560    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jxj\" (UniqueName: \"kubernetes.io/projected/322d2b78-70ae-408e-b76f-5a2d3b399319-kube-api-access-z6jxj\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608585    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/322d2b78-70ae-408e-b76f-5a2d3b399319-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608596    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/322d2b78-70ae-408e-b76f-5a2d3b399319-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608604    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjwbf\" (UniqueName: \"kubernetes.io/projected/58f6533a-0bbc-4899-9556-1fb69d45137f-kube-api-access-gjwbf\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608615    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwkz7\" (UniqueName: \"kubernetes.io/projected/7410d9f7-2e12-4514-a168-5485f7c76558-kube-api-access-dwkz7\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608623    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7410d9f7-2e12-4514-a168-5485f7c76558-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:08.608641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:08.608631    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f6533a-0bbc-4899-9556-1fb69d45137f-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:07:09.204390 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.204358    2568 generic.go:356] "Generic (PLEG): container finished" podID="7410d9f7-2e12-4514-a168-5485f7c76558" containerID="d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4" exitCode=0
Jul 27 20:07:09.204884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.204402    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"
Jul 27 20:07:09.204884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.204442    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8" event={"ID":"7410d9f7-2e12-4514-a168-5485f7c76558","Type":"ContainerDied","Data":"d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4"}
Jul 27 20:07:09.204884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.204492    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8" event={"ID":"7410d9f7-2e12-4514-a168-5485f7c76558","Type":"ContainerDied","Data":"f2e95c1493754213835c61c2848c5babe18f5e5407838d2cdc40bfcfcc625a50"}
Jul 27 20:07:09.204884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.204541    2568 scope.go:117] "RemoveContainer" containerID="d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4"
Jul 27 20:07:09.205937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.205721    2568 generic.go:356] "Generic (PLEG): container finished" podID="322d2b78-70ae-408e-b76f-5a2d3b399319" containerID="007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1" exitCode=0
Jul 27 20:07:09.205937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.205781    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"
Jul 27 20:07:09.205937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.205804    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" event={"ID":"322d2b78-70ae-408e-b76f-5a2d3b399319","Type":"ContainerDied","Data":"007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1"}
Jul 27 20:07:09.205937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.205857    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c" event={"ID":"322d2b78-70ae-408e-b76f-5a2d3b399319","Type":"ContainerDied","Data":"8a75a27413cca98906d581e4a0b94134b6a107ac5e8fd771bb3a7ddd00edac02"}
Jul 27 20:07:09.207379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.207359    2568 generic.go:356] "Generic (PLEG): container finished" podID="58f6533a-0bbc-4899-9556-1fb69d45137f" containerID="5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8" exitCode=0
Jul 27 20:07:09.207488 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.207402    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"
Jul 27 20:07:09.207488 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.207459    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb" event={"ID":"58f6533a-0bbc-4899-9556-1fb69d45137f","Type":"ContainerDied","Data":"5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8"}
Jul 27 20:07:09.207610 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.207507    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb" event={"ID":"58f6533a-0bbc-4899-9556-1fb69d45137f","Type":"ContainerDied","Data":"d5a1c48eaeb8855e07a956c0e35df737b3b58ed081a7484872095b68b3e361d0"}
Jul 27 20:07:09.216867 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.216850    2568 scope.go:117] "RemoveContainer" containerID="d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4"
Jul 27 20:07:09.217189 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:09.217165    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4\": container with ID starting with d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4 not found: ID does not exist" containerID="d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4"
Jul 27 20:07:09.217307 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.217194    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4"} err="failed to get container status \"d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4\": rpc error: code = NotFound desc = could not find container \"d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4\": container with ID starting with d088d289d202df5de9eda41456b3b4aa54047e35771caf58cb148953eae9afa4 not found: ID does not exist"
Jul 27 20:07:09.217307 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.217211    2568 scope.go:117] "RemoveContainer" containerID="007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1"
Jul 27 20:07:09.225800 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.225776    2568 scope.go:117] "RemoveContainer" containerID="007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1"
Jul 27 20:07:09.226026 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:09.226008    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1\": container with ID starting with 007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1 not found: ID does not exist" containerID="007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1"
Jul 27 20:07:09.226082 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.226033    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1"} err="failed to get container status \"007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1\": rpc error: code = NotFound desc = could not find container \"007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1\": container with ID starting with 007a8327dbef736a43944dd9b21a32a894ee169e0bfe4e1740b319536d343ab1 not found: ID does not exist"
Jul 27 20:07:09.226082 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.226050    2568 scope.go:117] "RemoveContainer" containerID="5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8"
Jul 27 20:07:09.233231 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.233209    2568 scope.go:117] "RemoveContainer" containerID="5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8"
Jul 27 20:07:09.233548 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:09.233511    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8\": container with ID starting with 5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8 not found: ID does not exist" containerID="5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8"
Jul 27 20:07:09.233677 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.233553    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8"} err="failed to get container status \"5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8\": rpc error: code = NotFound desc = could not find container \"5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8\": container with ID starting with 5078c617cda44a8e4dd52dfa6d88911e67daaa66909e1d9b163204547b5d8ff8 not found: ID does not exist"
Jul 27 20:07:09.235559 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.235540    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"]
Jul 27 20:07:09.240722 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.240701    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-554d51aa-77c7c55476-6z7b8"]
Jul 27 20:07:09.255022 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.254998    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"]
Jul 27 20:07:09.260242 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.260223    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-554d51aa-7cd96b6c64-x62qb"]
Jul 27 20:07:09.274911 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.274887    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"]
Jul 27 20:07:09.280078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.280049    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-554d51aa-588f7887f8-vb46c"]
Jul 27 20:07:09.567079 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.567040    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322d2b78-70ae-408e-b76f-5a2d3b399319" path="/var/lib/kubelet/pods/322d2b78-70ae-408e-b76f-5a2d3b399319/volumes"
Jul 27 20:07:09.567466 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.567447    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f6533a-0bbc-4899-9556-1fb69d45137f" path="/var/lib/kubelet/pods/58f6533a-0bbc-4899-9556-1fb69d45137f/volumes"
Jul 27 20:07:09.567852 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:09.567833    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7410d9f7-2e12-4514-a168-5485f7c76558" path="/var/lib/kubelet/pods/7410d9f7-2e12-4514-a168-5485f7c76558/volumes"
Jul 27 20:07:12.681404 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:12.681372    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:12.699180 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:12.699156    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"]
Jul 27 20:07:12.704084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:12.704064    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-554d51aa-4t75j"]
Jul 27 20:07:12.870104 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:12.870072    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:13.107925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:13.107891    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:13.347054 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:13.347021    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:13.567719 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:13.567685    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d727ea9-cd93-4d2b-a01e-968b9fddcbad" path="/var/lib/kubelet/pods/4d727ea9-cd93-4d2b-a01e-968b9fddcbad/volumes"
Jul 27 20:07:13.667685 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:13.667652    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:13.864134 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:13.864050    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:15.730494 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.730464    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:15.831206 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.831174    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:15.858469 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858440    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"]
Jul 27 20:07:15.858824 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858810    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f6533a-0bbc-4899-9556-1fb69d45137f" containerName="payload-processing"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858826    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f6533a-0bbc-4899-9556-1fb69d45137f" containerName="payload-processing"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858839    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d727ea9-cd93-4d2b-a01e-968b9fddcbad" containerName="revoke-keys"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858845    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d727ea9-cd93-4d2b-a01e-968b9fddcbad" containerName="revoke-keys"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858852    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322d2b78-70ae-408e-b76f-5a2d3b399319" containerName="maas-api"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858857    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="322d2b78-70ae-408e-b76f-5a2d3b399319" containerName="maas-api"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858865    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7410d9f7-2e12-4514-a168-5485f7c76558" containerName="payload-pre-processing"
Jul 27 20:07:15.858871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858872    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7410d9f7-2e12-4514-a168-5485f7c76558" containerName="payload-pre-processing"
Jul 27 20:07:15.859084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858923    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7410d9f7-2e12-4514-a168-5485f7c76558" containerName="payload-pre-processing"
Jul 27 20:07:15.859084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858934    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d727ea9-cd93-4d2b-a01e-968b9fddcbad" containerName="revoke-keys"
Jul 27 20:07:15.859084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858940    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f6533a-0bbc-4899-9556-1fb69d45137f" containerName="payload-processing"
Jul 27 20:07:15.859084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.858947    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="322d2b78-70ae-408e-b76f-5a2d3b399319" containerName="maas-api"
Jul 27 20:07:15.863604 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.863588    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:15.866662 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.866639    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a56cb698-serving-cert\""
Jul 27 20:07:15.872342 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.872320    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"]
Jul 27 20:07:15.956734 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.956686    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:15.972991 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.972964    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a89a9c-542d-48e4-93a9-06260a06669f-openshift-service-ca\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:15.973108 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.973002    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7ht\" (UniqueName: \"kubernetes.io/projected/58a89a9c-542d-48e4-93a9-06260a06669f-kube-api-access-hj7ht\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:15.973108 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:15.973041    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.073889 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.073864    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a89a9c-542d-48e4-93a9-06260a06669f-openshift-service-ca\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.074051 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.073904    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7ht\" (UniqueName: \"kubernetes.io/projected/58a89a9c-542d-48e4-93a9-06260a06669f-kube-api-access-hj7ht\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.074051 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.073949    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.074141 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:16.074061    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-serving-cert: secret "maas-api-e2e-derive-a56cb698-serving-cert" not found
Jul 27 20:07:16.074141 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:16.074130    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls podName:58a89a9c-542d-48e4-93a9-06260a06669f nodeName:}" failed. No retries permitted until 2026-07-27 20:07:16.574108666 +0000 UTC m=+2585.569956624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls") pod "maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" (UID: "58a89a9c-542d-48e4-93a9-06260a06669f") : secret "maas-api-e2e-derive-a56cb698-serving-cert" not found
Jul 27 20:07:16.074588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.074567    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a89a9c-542d-48e4-93a9-06260a06669f-openshift-service-ca\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.083286 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.083263    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7ht\" (UniqueName: \"kubernetes.io/projected/58a89a9c-542d-48e4-93a9-06260a06669f-kube-api-access-hj7ht\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.561395 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.561361    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:16.577783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.577754    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:16.577951 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:16.577901    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-serving-cert: secret "maas-api-e2e-derive-a56cb698-serving-cert" not found
Jul 27 20:07:16.578023 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:07:16.577960    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls podName:58a89a9c-542d-48e4-93a9-06260a06669f nodeName:}" failed. No retries permitted until 2026-07-27 20:07:17.577945067 +0000 UTC m=+2586.573793022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls") pod "maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" (UID: "58a89a9c-542d-48e4-93a9-06260a06669f") : secret "maas-api-e2e-derive-a56cb698-serving-cert" not found
Jul 27 20:07:16.659707 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.659675    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:16.674324 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.674294    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"]
Jul 27 20:07:16.677819 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.677805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.680788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.680751    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a56cb698-dockercfg-x9ljv\""
Jul 27 20:07:16.680907 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.680794    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a56cb698\""
Jul 27 20:07:16.688973 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.688947    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"]
Jul 27 20:07:16.705130 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.705104    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"]
Jul 27 20:07:16.708979 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.708963    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.719755 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.719731    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"]
Jul 27 20:07:16.779097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.779061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svd2b\" (UniqueName: \"kubernetes.io/projected/46badbd7-cd05-4996-9869-929ea99339cf-kube-api-access-svd2b\") pod \"payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") " pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.779458 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.779138    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmdz\" (UniqueName: \"kubernetes.io/projected/a1d3e169-ba3e-4103-996a-ec22afc6c718-kube-api-access-qnmdz\") pod \"payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.779458 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.779176    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3e169-ba3e-4103-996a-ec22afc6c718-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.779458 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.779197    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46badbd7-cd05-4996-9869-929ea99339cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") " pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.880673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.880587    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmdz\" (UniqueName: \"kubernetes.io/projected/a1d3e169-ba3e-4103-996a-ec22afc6c718-kube-api-access-qnmdz\") pod \"payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.880673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.880644    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3e169-ba3e-4103-996a-ec22afc6c718-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.880673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.880669    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46badbd7-cd05-4996-9869-929ea99339cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") " pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.880975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.880705    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svd2b\" (UniqueName: \"kubernetes.io/projected/46badbd7-cd05-4996-9869-929ea99339cf-kube-api-access-svd2b\") pod \"payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") " pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.881361 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.881338    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46badbd7-cd05-4996-9869-929ea99339cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") " pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.881446 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.881383    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3e169-ba3e-4103-996a-ec22afc6c718-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.905428 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.905401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmdz\" (UniqueName: \"kubernetes.io/projected/a1d3e169-ba3e-4103-996a-ec22afc6c718-kube-api-access-qnmdz\") pod \"payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:16.908600 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.908571    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svd2b\" (UniqueName: \"kubernetes.io/projected/46badbd7-cd05-4996-9869-929ea99339cf-kube-api-access-svd2b\") pod \"payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") " pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:16.987473 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.987438    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:07:16.989398 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:16.989382    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:17.022196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.022164    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:17.148246 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.148219    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"]
Jul 27 20:07:17.150991 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:07:17.150960    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d3e169_ba3e_4103_996a_ec22afc6c718.slice/crio-9e8d0b33bffeb4275c6c1e7f0be227470cf464bc84248de9a87f56aa2993504f WatchSource:0}: Error finding container 9e8d0b33bffeb4275c6c1e7f0be227470cf464bc84248de9a87f56aa2993504f: Status 404 returned error can't find the container with id 9e8d0b33bffeb4275c6c1e7f0be227470cf464bc84248de9a87f56aa2993504f
Jul 27 20:07:17.193625 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.187705    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"]
Jul 27 20:07:17.249442 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.249412    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx" event={"ID":"a1d3e169-ba3e-4103-996a-ec22afc6c718","Type":"ContainerStarted","Data":"9347e217d2501b7c480769b6d8b2c945051a0803f30fd068882a02cdfa1050ec"}
Jul 27 20:07:17.249603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.249475    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx" event={"ID":"a1d3e169-ba3e-4103-996a-ec22afc6c718","Type":"ContainerStarted","Data":"9e8d0b33bffeb4275c6c1e7f0be227470cf464bc84248de9a87f56aa2993504f"}
Jul 27 20:07:17.249603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.249508    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:17.251037 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.251014    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp" event={"ID":"46badbd7-cd05-4996-9869-929ea99339cf","Type":"ContainerStarted","Data":"8dbb3a0a09f3d6d3661d51e2d575a9fe75fca71d98a2c7febc384140c5c7b835"}
Jul 27 20:07:17.269924 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.269869    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx" podStartSLOduration=1.26985436 podStartE2EDuration="1.26985436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:17.268408968 +0000 UTC m=+2586.264256962" watchObservedRunningTime="2026-07-27 20:07:17.26985436 +0000 UTC m=+2586.265702357"
Jul 27 20:07:17.587316 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.587279    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:17.590057 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.590031    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") pod \"maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:17.674729 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.674681    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:17.819692 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.819668    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"]
Jul 27 20:07:17.822132 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:07:17.822103    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a89a9c_542d_48e4_93a9_06260a06669f.slice/crio-be5c10a3531c4adb44e39c2fa7368cb6f845194d2fda2e9c322da935dbfa891e WatchSource:0}: Error finding container be5c10a3531c4adb44e39c2fa7368cb6f845194d2fda2e9c322da935dbfa891e: Status 404 returned error can't find the container with id be5c10a3531c4adb44e39c2fa7368cb6f845194d2fda2e9c322da935dbfa891e
Jul 27 20:07:17.823487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:17.823470    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:07:18.259632 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:18.259595    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp" event={"ID":"46badbd7-cd05-4996-9869-929ea99339cf","Type":"ContainerStarted","Data":"02e344f3497516fe390e0eabf4856e521e6a4adb8acce1b31e822bcad64c17e9"}
Jul 27 20:07:18.259807 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:18.259771    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:07:18.261028 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:18.260997    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" event={"ID":"58a89a9c-542d-48e4-93a9-06260a06669f","Type":"ContainerStarted","Data":"be5c10a3531c4adb44e39c2fa7368cb6f845194d2fda2e9c322da935dbfa891e"}
Jul 27 20:07:18.279394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:18.279348    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp" podStartSLOduration=2.27933445 podStartE2EDuration="2.27933445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:18.277838313 +0000 UTC m=+2587.273686311" watchObservedRunningTime="2026-07-27 20:07:18.27933445 +0000 UTC m=+2587.275182425"
Jul 27 20:07:19.267129 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:19.267084    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" event={"ID":"58a89a9c-542d-48e4-93a9-06260a06669f","Type":"ContainerStarted","Data":"7cc5ab4d2b85c23e6f936269607becf71717a1e495b9741dbf180399c7fb6d26"}
Jul 27 20:07:19.267593 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:19.267402    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:19.289119 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:19.289065    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" podStartSLOduration=3.807750745 podStartE2EDuration="4.289050268s" totalImagesPullingTime="481.299523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:07:19.28777786 +0000 UTC m=+2588.283625835" watchObservedRunningTime="2026-07-27 20:07:19.289050268 +0000 UTC m=+2588.284898280"
Jul 27 20:07:25.278629 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:25.278591    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:07:28.263498 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:28.263464    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:07:29.270007 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:07:29.269974    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:08:11.752489 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:08:11.752448    2568 scope.go:117] "RemoveContainer" containerID="23578552d88bd310a76fb8abdea0e6384092f6e4a654e04eb7acfd92f962439d"
Jul 27 20:09:10.061770 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:09:10.061734    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:09:11.619824 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:09:11.619799    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:09:11.622136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:09:11.622104    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:11:36.190123 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:11:36.190081    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:11:37.469337 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:11:37.469296    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:22.037813 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.037775    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"]
Jul 27 20:12:22.041137 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.041118    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:22.043985 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.043963    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:12:22.057349 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.057321    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"]
Jul 27 20:12:22.167977 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.167942    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204133ab-ea78-470d-b281-3700e485e353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf\" (UID: \"204133ab-ea78-470d-b281-3700e485e353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:22.268693 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.268639    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204133ab-ea78-470d-b281-3700e485e353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf\" (UID: \"204133ab-ea78-470d-b281-3700e485e353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:22.269315 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.269288    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204133ab-ea78-470d-b281-3700e485e353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf\" (UID: \"204133ab-ea78-470d-b281-3700e485e353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:22.351312 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.351227    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:22.491083 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.491060    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"]
Jul 27 20:12:22.493463 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:12:22.493431    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204133ab_ea78_470d_b281_3700e485e353.slice/crio-77426ee6f94e5923992fbb3c1f44ef74d88664e94b37c7454452f24ae9d93cb9 WatchSource:0}: Error finding container 77426ee6f94e5923992fbb3c1f44ef74d88664e94b37c7454452f24ae9d93cb9: Status 404 returned error can't find the container with id 77426ee6f94e5923992fbb3c1f44ef74d88664e94b37c7454452f24ae9d93cb9
Jul 27 20:12:22.712677 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:22.712643    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf" event={"ID":"204133ab-ea78-470d-b281-3700e485e353","Type":"ContainerStarted","Data":"77426ee6f94e5923992fbb3c1f44ef74d88664e94b37c7454452f24ae9d93cb9"}
Jul 27 20:12:23.719458 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:23.719428    2568 generic.go:356] "Generic (PLEG): container finished" podID="204133ab-ea78-470d-b281-3700e485e353" containerID="436cdf971dd2e53cc5431be737f92360c0fb7e2528afc200c6ed7c7bd44409ba" exitCode=0
Jul 27 20:12:23.719875 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:23.719518    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf" event={"ID":"204133ab-ea78-470d-b281-3700e485e353","Type":"ContainerDied","Data":"436cdf971dd2e53cc5431be737f92360c0fb7e2528afc200c6ed7c7bd44409ba"}
Jul 27 20:12:24.856824 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:24.856800    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:24.895348 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:24.895319    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204133ab-ea78-470d-b281-3700e485e353-maas-api-service-ca\") pod \"204133ab-ea78-470d-b281-3700e485e353\" (UID: \"204133ab-ea78-470d-b281-3700e485e353\") "
Jul 27 20:12:24.895711 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:24.895689    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204133ab-ea78-470d-b281-3700e485e353-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "204133ab-ea78-470d-b281-3700e485e353" (UID: "204133ab-ea78-470d-b281-3700e485e353"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:24.996746 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:24.996656    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/204133ab-ea78-470d-b281-3700e485e353-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:25.731801 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:25.731748    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf" event={"ID":"204133ab-ea78-470d-b281-3700e485e353","Type":"ContainerDied","Data":"77426ee6f94e5923992fbb3c1f44ef74d88664e94b37c7454452f24ae9d93cb9"}
Jul 27 20:12:25.731987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:25.731857    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77426ee6f94e5923992fbb3c1f44ef74d88664e94b37c7454452f24ae9d93cb9"
Jul 27 20:12:25.731987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:25.731780    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"
Jul 27 20:12:26.222476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:26.222439    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:32.527821 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.527784    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"]
Jul 27 20:12:32.528176 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.528000    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" podUID="58a89a9c-542d-48e4-93a9-06260a06669f" containerName="maas-api" containerID="cri-o://7cc5ab4d2b85c23e6f936269607becf71717a1e495b9741dbf180399c7fb6d26" gracePeriod=30
Jul 27 20:12:32.554464 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.554435    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:32.585938 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.585768    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"]
Jul 27 20:12:32.586148 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.586020    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp" podUID="46badbd7-cd05-4996-9869-929ea99339cf" containerName="payload-processing" containerID="cri-o://02e344f3497516fe390e0eabf4856e521e6a4adb8acce1b31e822bcad64c17e9" gracePeriod=30
Jul 27 20:12:32.595038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.594879    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"]
Jul 27 20:12:32.595128 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.595099    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx" podUID="a1d3e169-ba3e-4103-996a-ec22afc6c718" containerName="payload-pre-processing" containerID="cri-o://9347e217d2501b7c480769b6d8b2c945051a0803f30fd068882a02cdfa1050ec" gracePeriod=30
Jul 27 20:12:32.634876 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.634837    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:32.738768 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.738724    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:32.768118 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.768070    2568 generic.go:356] "Generic (PLEG): container finished" podID="a1d3e169-ba3e-4103-996a-ec22afc6c718" containerID="9347e217d2501b7c480769b6d8b2c945051a0803f30fd068882a02cdfa1050ec" exitCode=0
Jul 27 20:12:32.768305 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.768141    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx" event={"ID":"a1d3e169-ba3e-4103-996a-ec22afc6c718","Type":"ContainerDied","Data":"9347e217d2501b7c480769b6d8b2c945051a0803f30fd068882a02cdfa1050ec"}
Jul 27 20:12:32.770021 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.770001    2568 generic.go:356] "Generic (PLEG): container finished" podID="58a89a9c-542d-48e4-93a9-06260a06669f" containerID="7cc5ab4d2b85c23e6f936269607becf71717a1e495b9741dbf180399c7fb6d26" exitCode=0
Jul 27 20:12:32.770135 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.770086    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" event={"ID":"58a89a9c-542d-48e4-93a9-06260a06669f","Type":"ContainerDied","Data":"7cc5ab4d2b85c23e6f936269607becf71717a1e495b9741dbf180399c7fb6d26"}
Jul 27 20:12:32.771516 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.771499    2568 generic.go:356] "Generic (PLEG): container finished" podID="46badbd7-cd05-4996-9869-929ea99339cf" containerID="02e344f3497516fe390e0eabf4856e521e6a4adb8acce1b31e822bcad64c17e9" exitCode=0
Jul 27 20:12:32.771583 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.771550    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp" event={"ID":"46badbd7-cd05-4996-9869-929ea99339cf","Type":"ContainerDied","Data":"02e344f3497516fe390e0eabf4856e521e6a4adb8acce1b31e822bcad64c17e9"}
Jul 27 20:12:32.868075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.868052    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:12:32.872758 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.872735    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:12:32.888121 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.888095    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:12:32.968900 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.968870    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46badbd7-cd05-4996-9869-929ea99339cf-plugins-config-volume\") pod \"46badbd7-cd05-4996-9869-929ea99339cf\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") "
Jul 27 20:12:32.968900 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.968908    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svd2b\" (UniqueName: \"kubernetes.io/projected/46badbd7-cd05-4996-9869-929ea99339cf-kube-api-access-svd2b\") pod \"46badbd7-cd05-4996-9869-929ea99339cf\" (UID: \"46badbd7-cd05-4996-9869-929ea99339cf\") "
Jul 27 20:12:32.969156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.968936    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnmdz\" (UniqueName: \"kubernetes.io/projected/a1d3e169-ba3e-4103-996a-ec22afc6c718-kube-api-access-qnmdz\") pod \"a1d3e169-ba3e-4103-996a-ec22afc6c718\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") "
Jul 27 20:12:32.969156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.968983    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj7ht\" (UniqueName: \"kubernetes.io/projected/58a89a9c-542d-48e4-93a9-06260a06669f-kube-api-access-hj7ht\") pod \"58a89a9c-542d-48e4-93a9-06260a06669f\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") "
Jul 27 20:12:32.969156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.969046    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") pod \"58a89a9c-542d-48e4-93a9-06260a06669f\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") "
Jul 27 20:12:32.969327 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.969203    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a89a9c-542d-48e4-93a9-06260a06669f-openshift-service-ca\") pod \"58a89a9c-542d-48e4-93a9-06260a06669f\" (UID: \"58a89a9c-542d-48e4-93a9-06260a06669f\") "
Jul 27 20:12:32.969327 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.969261    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3e169-ba3e-4103-996a-ec22afc6c718-plugins-config-volume\") pod \"a1d3e169-ba3e-4103-996a-ec22afc6c718\" (UID: \"a1d3e169-ba3e-4103-996a-ec22afc6c718\") "
Jul 27 20:12:32.969863 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.969612    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a89a9c-542d-48e4-93a9-06260a06669f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58a89a9c-542d-48e4-93a9-06260a06669f" (UID: "58a89a9c-542d-48e4-93a9-06260a06669f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:32.969863 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.969660    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d3e169-ba3e-4103-996a-ec22afc6c718-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1d3e169-ba3e-4103-996a-ec22afc6c718" (UID: "a1d3e169-ba3e-4103-996a-ec22afc6c718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:32.969863 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.969760    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46badbd7-cd05-4996-9869-929ea99339cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46badbd7-cd05-4996-9869-929ea99339cf" (UID: "46badbd7-cd05-4996-9869-929ea99339cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:12:32.971311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.971275    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46badbd7-cd05-4996-9869-929ea99339cf-kube-api-access-svd2b" (OuterVolumeSpecName: "kube-api-access-svd2b") pod "46badbd7-cd05-4996-9869-929ea99339cf" (UID: "46badbd7-cd05-4996-9869-929ea99339cf"). InnerVolumeSpecName "kube-api-access-svd2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:32.971311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.971281    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a89a9c-542d-48e4-93a9-06260a06669f-kube-api-access-hj7ht" (OuterVolumeSpecName: "kube-api-access-hj7ht") pod "58a89a9c-542d-48e4-93a9-06260a06669f" (UID: "58a89a9c-542d-48e4-93a9-06260a06669f"). InnerVolumeSpecName "kube-api-access-hj7ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:32.971677 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.971658    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58a89a9c-542d-48e4-93a9-06260a06669f" (UID: "58a89a9c-542d-48e4-93a9-06260a06669f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:12:32.971761 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:32.971670    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d3e169-ba3e-4103-996a-ec22afc6c718-kube-api-access-qnmdz" (OuterVolumeSpecName: "kube-api-access-qnmdz") pod "a1d3e169-ba3e-4103-996a-ec22afc6c718" (UID: "a1d3e169-ba3e-4103-996a-ec22afc6c718"). InnerVolumeSpecName "kube-api-access-qnmdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071035    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a89a9c-542d-48e4-93a9-06260a06669f-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071078    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a89a9c-542d-48e4-93a9-06260a06669f-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071088    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3e169-ba3e-4103-996a-ec22afc6c718-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071097    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46badbd7-cd05-4996-9869-929ea99339cf-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071107    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svd2b\" (UniqueName: \"kubernetes.io/projected/46badbd7-cd05-4996-9869-929ea99339cf-kube-api-access-svd2b\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071117    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmdz\" (UniqueName: \"kubernetes.io/projected/a1d3e169-ba3e-4103-996a-ec22afc6c718-kube-api-access-qnmdz\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.071133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.071125    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7ht\" (UniqueName: \"kubernetes.io/projected/58a89a9c-542d-48e4-93a9-06260a06669f-kube-api-access-hj7ht\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:12:33.777661 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.777626    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp" event={"ID":"46badbd7-cd05-4996-9869-929ea99339cf","Type":"ContainerDied","Data":"8dbb3a0a09f3d6d3661d51e2d575a9fe75fca71d98a2c7febc384140c5c7b835"}
Jul 27 20:12:33.778095 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.777663    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"
Jul 27 20:12:33.778095 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.777700    2568 scope.go:117] "RemoveContainer" containerID="02e344f3497516fe390e0eabf4856e521e6a4adb8acce1b31e822bcad64c17e9"
Jul 27 20:12:33.779136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.779109    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"
Jul 27 20:12:33.779224 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.779155    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx" event={"ID":"a1d3e169-ba3e-4103-996a-ec22afc6c718","Type":"ContainerDied","Data":"9e8d0b33bffeb4275c6c1e7f0be227470cf464bc84248de9a87f56aa2993504f"}
Jul 27 20:12:33.780755 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.780734    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"
Jul 27 20:12:33.780841 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.780731    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl" event={"ID":"58a89a9c-542d-48e4-93a9-06260a06669f","Type":"ContainerDied","Data":"be5c10a3531c4adb44e39c2fa7368cb6f845194d2fda2e9c322da935dbfa891e"}
Jul 27 20:12:33.787367 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.787352    2568 scope.go:117] "RemoveContainer" containerID="9347e217d2501b7c480769b6d8b2c945051a0803f30fd068882a02cdfa1050ec"
Jul 27 20:12:33.795740 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.795723    2568 scope.go:117] "RemoveContainer" containerID="7cc5ab4d2b85c23e6f936269607becf71717a1e495b9741dbf180399c7fb6d26"
Jul 27 20:12:33.808969 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.808947    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"]
Jul 27 20:12:33.815438 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.815417    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a56cb698-5d94fb86d6-9n5cp"]
Jul 27 20:12:33.828975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.828947    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"]
Jul 27 20:12:33.834917 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.834897    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a56cb698-6ff4687f89-td2cl"]
Jul 27 20:12:33.848010 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.847983    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"]
Jul 27 20:12:33.853382 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:33.853361    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a56cb698-79459665b4-pcbmx"]
Jul 27 20:12:35.567878 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:35.567845    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46badbd7-cd05-4996-9869-929ea99339cf" path="/var/lib/kubelet/pods/46badbd7-cd05-4996-9869-929ea99339cf/volumes"
Jul 27 20:12:35.568244 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:35.568204    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a89a9c-542d-48e4-93a9-06260a06669f" path="/var/lib/kubelet/pods/58a89a9c-542d-48e4-93a9-06260a06669f/volumes"
Jul 27 20:12:35.568497 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:35.568484    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d3e169-ba3e-4103-996a-ec22afc6c718" path="/var/lib/kubelet/pods/a1d3e169-ba3e-4103-996a-ec22afc6c718/volumes"
Jul 27 20:12:37.119666 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:37.119621    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:37.144376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:37.144334    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"]
Jul 27 20:12:37.147747 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:37.147718    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a56cb698-fl7vf"]
Jul 27 20:12:37.292977 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:37.292937    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:37.534358 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:37.534311    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:37.567083 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:37.567054    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="204133ab-ea78-470d-b281-3700e485e353" path="/var/lib/kubelet/pods/204133ab-ea78-470d-b281-3700e485e353/volumes"
Jul 27 20:12:38.000408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:38.000312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:38.100678 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:38.100643    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:38.200914 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:38.200875    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:39.612735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:39.612698    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:46.569559 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:46.569504    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:46.673790 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:46.673753    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:46.769425 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:46.769387    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:46.830311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:46.830204    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:12:47.260562 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:12:47.260457    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:11.800698 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:11.800666    2568 scope.go:117] "RemoveContainer" containerID="da9cd8b0c85b87f7078f4f0716e82767fce68930ab6ce4b5e7cb5194009f8c15"
Jul 27 20:13:13.194274 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:13.194235    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:20.949236 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:20.949187    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:21.016905 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:21.016867    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:21.107267 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:21.107228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:21.204425 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:21.204342    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:36.661912 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:36.661867    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:37.070897 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:37.070863    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:43.532955 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:43.532916    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:43.609917 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:43.609884    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:43.675967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:43.675930    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:43.776028 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:43.775996    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:54.255082 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:54.255050    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:13:55.011209 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:13:55.011167    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:02.260827 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:02.260792    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:02.362744 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:02.362715    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:02.461885 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:02.461838    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:02.562577 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:02.562513    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:08.419540 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:08.419488    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:08.533737 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:08.533706    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:08.622908 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:08.622864    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:10.200723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:10.200688    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:11.647459 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:11.647434    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:14:11.651325 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:11.651296    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:14:17.014487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:17.014446    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:17.114690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:17.114654    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:17.683876 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:17.683835    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:24.040286 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:24.040252    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:24.098900 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:24.098865    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:24.276311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:24.276278    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:24.578503 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:24.578463    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:25.086922 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:25.086884    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:27.280207 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.280172    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:27.376475 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.376447    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:27.385390 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385361    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"]
Jul 27 20:14:27.385728 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385715    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204133ab-ea78-470d-b281-3700e485e353" containerName="revoke-keys"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385730    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="204133ab-ea78-470d-b281-3700e485e353" containerName="revoke-keys"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385740    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d3e169-ba3e-4103-996a-ec22afc6c718" containerName="payload-pre-processing"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385746    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d3e169-ba3e-4103-996a-ec22afc6c718" containerName="payload-pre-processing"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385762    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46badbd7-cd05-4996-9869-929ea99339cf" containerName="payload-processing"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385768    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="46badbd7-cd05-4996-9869-929ea99339cf" containerName="payload-processing"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385776    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a89a9c-542d-48e4-93a9-06260a06669f" containerName="maas-api"
Jul 27 20:14:27.385783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385780    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a89a9c-542d-48e4-93a9-06260a06669f" containerName="maas-api"
Jul 27 20:14:27.385987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385836    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="46badbd7-cd05-4996-9869-929ea99339cf" containerName="payload-processing"
Jul 27 20:14:27.385987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385843    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a89a9c-542d-48e4-93a9-06260a06669f" containerName="maas-api"
Jul 27 20:14:27.385987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385850    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d3e169-ba3e-4103-996a-ec22afc6c718" containerName="payload-pre-processing"
Jul 27 20:14:27.385987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.385857    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="204133ab-ea78-470d-b281-3700e485e353" containerName="revoke-keys"
Jul 27 20:14:27.388754 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.388737    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.389382 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.389363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxz9k\" (UniqueName: \"kubernetes.io/projected/5c4d1f95-648c-466b-87fc-08657d6e710f-kube-api-access-sxz9k\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.389459 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.389448    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4d1f95-648c-466b-87fc-08657d6e710f-openshift-service-ca\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.389494 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.389477    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.391539 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.391501    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0197ac0c-serving-cert\""
Jul 27 20:14:27.402491 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.402464    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"]
Jul 27 20:14:27.474775 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.474738    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:27.489928 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.489903    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.490087 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.489943    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxz9k\" (UniqueName: \"kubernetes.io/projected/5c4d1f95-648c-466b-87fc-08657d6e710f-kube-api-access-sxz9k\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.490087 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.490000    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4d1f95-648c-466b-87fc-08657d6e710f-openshift-service-ca\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.490087 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:27.490067    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-serving-cert: secret "maas-api-e2e-mt-0197ac0c-serving-cert" not found
Jul 27 20:14:27.490190 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:27.490134    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls podName:5c4d1f95-648c-466b-87fc-08657d6e710f nodeName:}" failed. No retries permitted until 2026-07-27 20:14:27.990114453 +0000 UTC m=+3016.985962410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls") pod "maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" (UID: "5c4d1f95-648c-466b-87fc-08657d6e710f") : secret "maas-api-e2e-mt-0197ac0c-serving-cert" not found
Jul 27 20:14:27.490667 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.490651    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4d1f95-648c-466b-87fc-08657d6e710f-openshift-service-ca\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.499996 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.499974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxz9k\" (UniqueName: \"kubernetes.io/projected/5c4d1f95-648c-466b-87fc-08657d6e710f-kube-api-access-sxz9k\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.994132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:27.994099    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:27.994322 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:27.994302    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-serving-cert: secret "maas-api-e2e-mt-0197ac0c-serving-cert" not found
Jul 27 20:14:27.994377 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:27.994368    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls podName:5c4d1f95-648c-466b-87fc-08657d6e710f nodeName:}" failed. No retries permitted until 2026-07-27 20:14:28.994353569 +0000 UTC m=+3017.990201524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls") pod "maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" (UID: "5c4d1f95-648c-466b-87fc-08657d6e710f") : secret "maas-api-e2e-mt-0197ac0c-serving-cert" not found
Jul 27 20:14:28.047769 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.047731    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:28.151092 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.151039    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:28.179869 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.179829    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"]
Jul 27 20:14:28.183448 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.183432    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.186614 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.186592    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0197ac0c-dockercfg-zwldc\""
Jul 27 20:14:28.186749 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.186599    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0197ac0c\""
Jul 27 20:14:28.195429 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.195403    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"]
Jul 27 20:14:28.195759 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.195741    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrwm\" (UniqueName: \"kubernetes.io/projected/0099058b-442b-472f-b31f-650c684c186d-kube-api-access-pgrwm\") pod \"payload-processing-e2e-mt-0197ac0c-569467646-g7kqh\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") " pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.195808 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.195783    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0099058b-442b-472f-b31f-650c684c186d-plugins-config-volume\") pod \"payload-processing-e2e-mt-0197ac0c-569467646-g7kqh\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") " pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.240875 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.240840    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"]
Jul 27 20:14:28.244187 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.244144    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.253724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.253700    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"]
Jul 27 20:14:28.297179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.297146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrwm\" (UniqueName: \"kubernetes.io/projected/0099058b-442b-472f-b31f-650c684c186d-kube-api-access-pgrwm\") pod \"payload-processing-e2e-mt-0197ac0c-569467646-g7kqh\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") " pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.297573 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.297192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlqr\" (UniqueName: \"kubernetes.io/projected/73047a3d-424e-4831-a180-c8fcb414ab9c-kube-api-access-lmlqr\") pod \"payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.297573 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.297283    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0099058b-442b-472f-b31f-650c684c186d-plugins-config-volume\") pod \"payload-processing-e2e-mt-0197ac0c-569467646-g7kqh\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") " pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.297573 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.297347    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73047a3d-424e-4831-a180-c8fcb414ab9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.298009 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.297990    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0099058b-442b-472f-b31f-650c684c186d-plugins-config-volume\") pod \"payload-processing-e2e-mt-0197ac0c-569467646-g7kqh\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") " pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.307632 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.307612    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrwm\" (UniqueName: \"kubernetes.io/projected/0099058b-442b-472f-b31f-650c684c186d-kube-api-access-pgrwm\") pod \"payload-processing-e2e-mt-0197ac0c-569467646-g7kqh\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") " pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.397928 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.397894    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73047a3d-424e-4831-a180-c8fcb414ab9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.398099 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.397975    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlqr\" (UniqueName: \"kubernetes.io/projected/73047a3d-424e-4831-a180-c8fcb414ab9c-kube-api-access-lmlqr\") pod \"payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.398562 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.398542    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73047a3d-424e-4831-a180-c8fcb414ab9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.410303 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.410278    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlqr\" (UniqueName: \"kubernetes.io/projected/73047a3d-424e-4831-a180-c8fcb414ab9c-kube-api-access-lmlqr\") pod \"payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.494904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.494807    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:28.555289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.555253    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:28.640630 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.640458    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"]
Jul 27 20:14:28.643030 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:14:28.642997    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0099058b_442b_472f_b31f_650c684c186d.slice/crio-d25b31947b645404d28adfe95c857ac3302b057163fbb3e7cacfbcc50cda64b0 WatchSource:0}: Error finding container d25b31947b645404d28adfe95c857ac3302b057163fbb3e7cacfbcc50cda64b0: Status 404 returned error can't find the container with id d25b31947b645404d28adfe95c857ac3302b057163fbb3e7cacfbcc50cda64b0
Jul 27 20:14:28.702960 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:28.702936    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"]
Jul 27 20:14:28.706227 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:14:28.706201    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73047a3d_424e_4831_a180_c8fcb414ab9c.slice/crio-c2481ad617330a07b9f08ba8e5c474da8ffce5f47861fa57789866f29c3457ee WatchSource:0}: Error finding container c2481ad617330a07b9f08ba8e5c474da8ffce5f47861fa57789866f29c3457ee: Status 404 returned error can't find the container with id c2481ad617330a07b9f08ba8e5c474da8ffce5f47861fa57789866f29c3457ee
Jul 27 20:14:29.005472 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.005378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:29.007999 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.007973    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") pod \"maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:29.201176 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.201135    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:29.323678 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.323645    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh" event={"ID":"0099058b-442b-472f-b31f-650c684c186d","Type":"ContainerStarted","Data":"64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a"}
Jul 27 20:14:29.324135 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.323823    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:29.324135 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.323847    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh" event={"ID":"0099058b-442b-472f-b31f-650c684c186d","Type":"ContainerStarted","Data":"d25b31947b645404d28adfe95c857ac3302b057163fbb3e7cacfbcc50cda64b0"}
Jul 27 20:14:29.325406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.325379    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh" event={"ID":"73047a3d-424e-4831-a180-c8fcb414ab9c","Type":"ContainerStarted","Data":"a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5"}
Jul 27 20:14:29.325544 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.325422    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh" event={"ID":"73047a3d-424e-4831-a180-c8fcb414ab9c","Type":"ContainerStarted","Data":"c2481ad617330a07b9f08ba8e5c474da8ffce5f47861fa57789866f29c3457ee"}
Jul 27 20:14:29.325544 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.325481    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:29.342565 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.342507    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"]
Jul 27 20:14:29.342695 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.342598    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh" podStartSLOduration=1.342583213 podStartE2EDuration="1.342583213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:29.341715592 +0000 UTC m=+3018.337563586" watchObservedRunningTime="2026-07-27 20:14:29.342583213 +0000 UTC m=+3018.338431192"
Jul 27 20:14:29.344921 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:14:29.344898    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4d1f95_648c_466b_87fc_08657d6e710f.slice/crio-23511cf4bc38a0085d28a1c2e426fed3623cf2d151d0c3275b3c9c4738bea24e WatchSource:0}: Error finding container 23511cf4bc38a0085d28a1c2e426fed3623cf2d151d0c3275b3c9c4738bea24e: Status 404 returned error can't find the container with id 23511cf4bc38a0085d28a1c2e426fed3623cf2d151d0c3275b3c9c4738bea24e
Jul 27 20:14:29.346286 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.346268    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:14:29.359116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:29.359075    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh" podStartSLOduration=1.359062388 podStartE2EDuration="1.359062388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:29.358449629 +0000 UTC m=+3018.354297618" watchObservedRunningTime="2026-07-27 20:14:29.359062388 +0000 UTC m=+3018.354910411"
Jul 27 20:14:30.331364 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:30.331323    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" event={"ID":"5c4d1f95-648c-466b-87fc-08657d6e710f","Type":"ContainerStarted","Data":"84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737"}
Jul 27 20:14:30.331771 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:30.331385    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" event={"ID":"5c4d1f95-648c-466b-87fc-08657d6e710f","Type":"ContainerStarted","Data":"23511cf4bc38a0085d28a1c2e426fed3623cf2d151d0c3275b3c9c4738bea24e"}
Jul 27 20:14:30.331771 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:30.331446    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:30.351139 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:30.351085    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" podStartSLOduration=2.905626237 podStartE2EDuration="3.351070107s" totalImagesPullingTime="445.44387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:30.350970793 +0000 UTC m=+3019.346818771" watchObservedRunningTime="2026-07-27 20:14:30.351070107 +0000 UTC m=+3019.346918084"
Jul 27 20:14:31.783728 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:31.783690    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:32.369438 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:32.369398    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:32.467919 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:32.467865    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:32.791336 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:32.791293    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:36.344354 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:36.344324    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:39.292984 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.292915    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:39.353720 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.353686    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:39.640451 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.640354    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"]
Jul 27 20:14:39.644242 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.644218    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:39.646945 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.646925    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:14:39.652649 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.652624    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"]
Jul 27 20:14:39.695322 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.695288    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421e283e-163d-4cf5-9982-494b9d90a689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9\" (UID: \"421e283e-163d-4cf5-9982-494b9d90a689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:39.796321 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.796290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421e283e-163d-4cf5-9982-494b9d90a689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9\" (UID: \"421e283e-163d-4cf5-9982-494b9d90a689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:39.796989 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.796965    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421e283e-163d-4cf5-9982-494b9d90a689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9\" (UID: \"421e283e-163d-4cf5-9982-494b9d90a689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:39.955150 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:39.955050    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:40.094495 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:40.094469    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"]
Jul 27 20:14:40.097166 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:14:40.097142    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421e283e_163d_4cf5_9982_494b9d90a689.slice/crio-c48f3d29353bcf5c811d0cccb42b953b8c4e90e31febe33cf57f28090f6c5533 WatchSource:0}: Error finding container c48f3d29353bcf5c811d0cccb42b953b8c4e90e31febe33cf57f28090f6c5533: Status 404 returned error can't find the container with id c48f3d29353bcf5c811d0cccb42b953b8c4e90e31febe33cf57f28090f6c5533
Jul 27 20:14:40.333320 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:40.333297    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:40.333747 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:40.333376    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:40.391598 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:40.391558    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9" event={"ID":"421e283e-163d-4cf5-9982-494b9d90a689","Type":"ContainerStarted","Data":"932f2764f8f3f1ceba6eb6f5a8d71e7896a4d03a3facc1f5a4d17f99bda9fde6"}
Jul 27 20:14:40.391770 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:40.391656    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9" event={"ID":"421e283e-163d-4cf5-9982-494b9d90a689","Type":"ContainerStarted","Data":"c48f3d29353bcf5c811d0cccb42b953b8c4e90e31febe33cf57f28090f6c5533"}
Jul 27 20:14:40.410915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:40.410865    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9" podStartSLOduration=1.4108486089999999 podStartE2EDuration="1.410848609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:14:40.409714957 +0000 UTC m=+3029.405562936" watchObservedRunningTime="2026-07-27 20:14:40.410848609 +0000 UTC m=+3029.406696612"
Jul 27 20:14:41.398297 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:41.398265    2568 generic.go:356] "Generic (PLEG): container finished" podID="421e283e-163d-4cf5-9982-494b9d90a689" containerID="932f2764f8f3f1ceba6eb6f5a8d71e7896a4d03a3facc1f5a4d17f99bda9fde6" exitCode=0
Jul 27 20:14:41.398679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:41.398357    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9" event={"ID":"421e283e-163d-4cf5-9982-494b9d90a689","Type":"ContainerDied","Data":"932f2764f8f3f1ceba6eb6f5a8d71e7896a4d03a3facc1f5a4d17f99bda9fde6"}
Jul 27 20:14:42.533425 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:42.533402    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:42.619574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:42.619507    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421e283e-163d-4cf5-9982-494b9d90a689-maas-api-service-ca\") pod \"421e283e-163d-4cf5-9982-494b9d90a689\" (UID: \"421e283e-163d-4cf5-9982-494b9d90a689\") "
Jul 27 20:14:42.619866 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:42.619836    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421e283e-163d-4cf5-9982-494b9d90a689-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "421e283e-163d-4cf5-9982-494b9d90a689" (UID: "421e283e-163d-4cf5-9982-494b9d90a689"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:14:42.720931 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:42.720850    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421e283e-163d-4cf5-9982-494b9d90a689-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:43.409486 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:43.409456    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"
Jul 27 20:14:43.409674 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:43.409452    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9" event={"ID":"421e283e-163d-4cf5-9982-494b9d90a689","Type":"ContainerDied","Data":"c48f3d29353bcf5c811d0cccb42b953b8c4e90e31febe33cf57f28090f6c5533"}
Jul 27 20:14:43.409674 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:43.409580    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48f3d29353bcf5c811d0cccb42b953b8c4e90e31febe33cf57f28090f6c5533"
Jul 27 20:14:49.793465 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.793433    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"]
Jul 27 20:14:49.793952 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.793691    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" podUID="5c4d1f95-648c-466b-87fc-08657d6e710f" containerName="maas-api" containerID="cri-o://84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737" gracePeriod=30
Jul 27 20:14:49.821847 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.821820    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:49.850110 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.850084    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"]
Jul 27 20:14:49.850352 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.850330    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh" podUID="0099058b-442b-472f-b31f-650c684c186d" containerName="payload-processing" containerID="cri-o://64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a" gracePeriod=30
Jul 27 20:14:49.859960 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.859917    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"]
Jul 27 20:14:49.860387 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.860355    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh" podUID="73047a3d-424e-4831-a180-c8fcb414ab9c" containerName="payload-pre-processing" containerID="cri-o://a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5" gracePeriod=30
Jul 27 20:14:49.885552 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.885505    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:49.986026 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:49.985989    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:50.111341 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.111312    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:50.116693 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.116658    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:50.154271 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.154249    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:50.190502 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.190472    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") pod \"5c4d1f95-648c-466b-87fc-08657d6e710f\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") "
Jul 27 20:14:50.190667 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.190593    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73047a3d-424e-4831-a180-c8fcb414ab9c-plugins-config-volume\") pod \"73047a3d-424e-4831-a180-c8fcb414ab9c\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") "
Jul 27 20:14:50.190667 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.190645    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4d1f95-648c-466b-87fc-08657d6e710f-openshift-service-ca\") pod \"5c4d1f95-648c-466b-87fc-08657d6e710f\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") "
Jul 27 20:14:50.190667 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.190665    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmlqr\" (UniqueName: \"kubernetes.io/projected/73047a3d-424e-4831-a180-c8fcb414ab9c-kube-api-access-lmlqr\") pod \"73047a3d-424e-4831-a180-c8fcb414ab9c\" (UID: \"73047a3d-424e-4831-a180-c8fcb414ab9c\") "
Jul 27 20:14:50.190823 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.190685    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxz9k\" (UniqueName: \"kubernetes.io/projected/5c4d1f95-648c-466b-87fc-08657d6e710f-kube-api-access-sxz9k\") pod \"5c4d1f95-648c-466b-87fc-08657d6e710f\" (UID: \"5c4d1f95-648c-466b-87fc-08657d6e710f\") "
Jul 27 20:14:50.191013 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.190979    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4d1f95-648c-466b-87fc-08657d6e710f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c4d1f95-648c-466b-87fc-08657d6e710f" (UID: "5c4d1f95-648c-466b-87fc-08657d6e710f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:14:50.191108 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.191010    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73047a3d-424e-4831-a180-c8fcb414ab9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73047a3d-424e-4831-a180-c8fcb414ab9c" (UID: "73047a3d-424e-4831-a180-c8fcb414ab9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:14:50.192943 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.192915    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c4d1f95-648c-466b-87fc-08657d6e710f" (UID: "5c4d1f95-648c-466b-87fc-08657d6e710f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:14:50.193054 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.192940    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73047a3d-424e-4831-a180-c8fcb414ab9c-kube-api-access-lmlqr" (OuterVolumeSpecName: "kube-api-access-lmlqr") pod "73047a3d-424e-4831-a180-c8fcb414ab9c" (UID: "73047a3d-424e-4831-a180-c8fcb414ab9c"). InnerVolumeSpecName "kube-api-access-lmlqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:50.193054 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.192940    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4d1f95-648c-466b-87fc-08657d6e710f-kube-api-access-sxz9k" (OuterVolumeSpecName: "kube-api-access-sxz9k") pod "5c4d1f95-648c-466b-87fc-08657d6e710f" (UID: "5c4d1f95-648c-466b-87fc-08657d6e710f"). InnerVolumeSpecName "kube-api-access-sxz9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:50.291514 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291482    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgrwm\" (UniqueName: \"kubernetes.io/projected/0099058b-442b-472f-b31f-650c684c186d-kube-api-access-pgrwm\") pod \"0099058b-442b-472f-b31f-650c684c186d\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") "
Jul 27 20:14:50.291712 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291575    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0099058b-442b-472f-b31f-650c684c186d-plugins-config-volume\") pod \"0099058b-442b-472f-b31f-650c684c186d\" (UID: \"0099058b-442b-472f-b31f-650c684c186d\") "
Jul 27 20:14:50.291788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291740    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxz9k\" (UniqueName: \"kubernetes.io/projected/5c4d1f95-648c-466b-87fc-08657d6e710f-kube-api-access-sxz9k\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.291788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291751    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4d1f95-648c-466b-87fc-08657d6e710f-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.291788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291760    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73047a3d-424e-4831-a180-c8fcb414ab9c-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.291788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291770    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4d1f95-648c-466b-87fc-08657d6e710f-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.291788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291778    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlqr\" (UniqueName: \"kubernetes.io/projected/73047a3d-424e-4831-a180-c8fcb414ab9c-kube-api-access-lmlqr\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.292003 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.291924    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0099058b-442b-472f-b31f-650c684c186d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0099058b-442b-472f-b31f-650c684c186d" (UID: "0099058b-442b-472f-b31f-650c684c186d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:14:50.293671 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.293652    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0099058b-442b-472f-b31f-650c684c186d-kube-api-access-pgrwm" (OuterVolumeSpecName: "kube-api-access-pgrwm") pod "0099058b-442b-472f-b31f-650c684c186d" (UID: "0099058b-442b-472f-b31f-650c684c186d"). InnerVolumeSpecName "kube-api-access-pgrwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:14:50.392906 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.392816    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0099058b-442b-472f-b31f-650c684c186d-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.392906 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.392845    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgrwm\" (UniqueName: \"kubernetes.io/projected/0099058b-442b-472f-b31f-650c684c186d-kube-api-access-pgrwm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:14:50.448665 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.448630    2568 generic.go:356] "Generic (PLEG): container finished" podID="0099058b-442b-472f-b31f-650c684c186d" containerID="64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a" exitCode=0
Jul 27 20:14:50.448864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.448676    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"
Jul 27 20:14:50.448864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.448719    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh" event={"ID":"0099058b-442b-472f-b31f-650c684c186d","Type":"ContainerDied","Data":"64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a"}
Jul 27 20:14:50.448864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.448777    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh" event={"ID":"0099058b-442b-472f-b31f-650c684c186d","Type":"ContainerDied","Data":"d25b31947b645404d28adfe95c857ac3302b057163fbb3e7cacfbcc50cda64b0"}
Jul 27 20:14:50.448864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.448806    2568 scope.go:117] "RemoveContainer" containerID="64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a"
Jul 27 20:14:50.450048 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.450025    2568 generic.go:356] "Generic (PLEG): container finished" podID="5c4d1f95-648c-466b-87fc-08657d6e710f" containerID="84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737" exitCode=0
Jul 27 20:14:50.450157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.450079    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"
Jul 27 20:14:50.450157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.450099    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" event={"ID":"5c4d1f95-648c-466b-87fc-08657d6e710f","Type":"ContainerDied","Data":"84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737"}
Jul 27 20:14:50.450157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.450142    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd" event={"ID":"5c4d1f95-648c-466b-87fc-08657d6e710f","Type":"ContainerDied","Data":"23511cf4bc38a0085d28a1c2e426fed3623cf2d151d0c3275b3c9c4738bea24e"}
Jul 27 20:14:50.451403 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.451383    2568 generic.go:356] "Generic (PLEG): container finished" podID="73047a3d-424e-4831-a180-c8fcb414ab9c" containerID="a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5" exitCode=0
Jul 27 20:14:50.451502 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.451445    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"
Jul 27 20:14:50.451502 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.451469    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh" event={"ID":"73047a3d-424e-4831-a180-c8fcb414ab9c","Type":"ContainerDied","Data":"a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5"}
Jul 27 20:14:50.451620 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.451515    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh" event={"ID":"73047a3d-424e-4831-a180-c8fcb414ab9c","Type":"ContainerDied","Data":"c2481ad617330a07b9f08ba8e5c474da8ffce5f47861fa57789866f29c3457ee"}
Jul 27 20:14:50.458125 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.458101    2568 scope.go:117] "RemoveContainer" containerID="64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a"
Jul 27 20:14:50.458394 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:50.458375    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a\": container with ID starting with 64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a not found: ID does not exist" containerID="64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a"
Jul 27 20:14:50.458487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.458401    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a"} err="failed to get container status \"64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a\": rpc error: code = NotFound desc = could not find container \"64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a\": container with ID starting with 64e30f486241a718d2d007a02a8a0ccf766e55c430987e2f14f8480a1bcc018a not found: ID does not exist"
Jul 27 20:14:50.458487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.458423    2568 scope.go:117] "RemoveContainer" containerID="84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737"
Jul 27 20:14:50.467393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.467379    2568 scope.go:117] "RemoveContainer" containerID="84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737"
Jul 27 20:14:50.467667 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:50.467646    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737\": container with ID starting with 84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737 not found: ID does not exist" containerID="84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737"
Jul 27 20:14:50.467735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.467673    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737"} err="failed to get container status \"84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737\": rpc error: code = NotFound desc = could not find container \"84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737\": container with ID starting with 84b2f75bca6427a57354f9ea79206b36944326b5389d76b3a2644f97f2b82737 not found: ID does not exist"
Jul 27 20:14:50.467735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.467689    2568 scope.go:117] "RemoveContainer" containerID="a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5"
Jul 27 20:14:50.475231 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.475215    2568 scope.go:117] "RemoveContainer" containerID="a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5"
Jul 27 20:14:50.475480 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:14:50.475460    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5\": container with ID starting with a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5 not found: ID does not exist" containerID="a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5"
Jul 27 20:14:50.475537 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.475486    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5"} err="failed to get container status \"a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5\": rpc error: code = NotFound desc = could not find container \"a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5\": container with ID starting with a908bfd0cf1d1be639751cce3dc42f686c0fa806e7891b68a92700e50db6cbc5 not found: ID does not exist"
Jul 27 20:14:50.478559 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.478500    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"]
Jul 27 20:14:50.481832 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.481812    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0197ac0c-6f4f997885-cr4rd"]
Jul 27 20:14:50.494935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.494905    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"]
Jul 27 20:14:50.499464 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.499441    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0197ac0c-569467646-g7kqh"]
Jul 27 20:14:50.512561 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.512536    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"]
Jul 27 20:14:50.519083 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:50.519060    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0197ac0c-f845dc956-v72qh"]
Jul 27 20:14:51.568494 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:51.568461    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0099058b-442b-472f-b31f-650c684c186d" path="/var/lib/kubelet/pods/0099058b-442b-472f-b31f-650c684c186d/volumes"
Jul 27 20:14:51.568881 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:51.568856    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4d1f95-648c-466b-87fc-08657d6e710f" path="/var/lib/kubelet/pods/5c4d1f95-648c-466b-87fc-08657d6e710f/volumes"
Jul 27 20:14:51.569158 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:51.569146    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73047a3d-424e-4831-a180-c8fcb414ab9c" path="/var/lib/kubelet/pods/73047a3d-424e-4831-a180-c8fcb414ab9c/volumes"
Jul 27 20:14:54.723405 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:54.723363    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:54.723908 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:54.723889    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"]
Jul 27 20:14:54.725453 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:54.725433    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0197ac0c-xbrn9"]
Jul 27 20:14:55.092756 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:55.092709    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:14:55.567106 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:14:55.567062    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="421e283e-163d-4cf5-9982-494b9d90a689" path="/var/lib/kubelet/pods/421e283e-163d-4cf5-9982-494b9d90a689/volumes"
Jul 27 20:15:00.141254 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141217    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"]
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141578    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4d1f95-648c-466b-87fc-08657d6e710f" containerName="maas-api"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141590    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4d1f95-648c-466b-87fc-08657d6e710f" containerName="maas-api"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141605    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099058b-442b-472f-b31f-650c684c186d" containerName="payload-processing"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141611    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099058b-442b-472f-b31f-650c684c186d" containerName="payload-processing"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141617    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73047a3d-424e-4831-a180-c8fcb414ab9c" containerName="payload-pre-processing"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141622    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="73047a3d-424e-4831-a180-c8fcb414ab9c" containerName="payload-pre-processing"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141628    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421e283e-163d-4cf5-9982-494b9d90a689" containerName="revoke-keys"
Jul 27 20:15:00.141652 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141633    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="421e283e-163d-4cf5-9982-494b9d90a689" containerName="revoke-keys"
Jul 27 20:15:00.141914 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141688    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4d1f95-648c-466b-87fc-08657d6e710f" containerName="maas-api"
Jul 27 20:15:00.141914 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141695    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="73047a3d-424e-4831-a180-c8fcb414ab9c" containerName="payload-pre-processing"
Jul 27 20:15:00.141914 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141703    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0099058b-442b-472f-b31f-650c684c186d" containerName="payload-processing"
Jul 27 20:15:00.141914 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.141709    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="421e283e-163d-4cf5-9982-494b9d90a689" containerName="revoke-keys"
Jul 27 20:15:00.146297 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.146282    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"
Jul 27 20:15:00.149351 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.149332    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:15:00.152254 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.152215    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"]
Jul 27 20:15:00.156870 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.156851    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"
Jul 27 20:15:00.296577 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.296542    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"]
Jul 27 20:15:00.302877 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:15:00.302837    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f108ba5_c3c7_458d_ad2a_98af57b83616.slice/crio-6ed0b4cd002adb7d0a82d96471d7686a8104e9976274b5f4e9989b3e4056b876 WatchSource:0}: Error finding container 6ed0b4cd002adb7d0a82d96471d7686a8104e9976274b5f4e9989b3e4056b876: Status 404 returned error can't find the container with id 6ed0b4cd002adb7d0a82d96471d7686a8104e9976274b5f4e9989b3e4056b876
Jul 27 20:15:00.502076 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.502045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b" event={"ID":"9f108ba5-c3c7-458d-ad2a-98af57b83616","Type":"ContainerStarted","Data":"5c271c360f850ff79398d0b84ffbd9dbcbe36ae56d344bfba06a6cbaca057614"}
Jul 27 20:15:00.502210 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.502097    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b" event={"ID":"9f108ba5-c3c7-458d-ad2a-98af57b83616","Type":"ContainerStarted","Data":"6ed0b4cd002adb7d0a82d96471d7686a8104e9976274b5f4e9989b3e4056b876"}
Jul 27 20:15:00.519124 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:00.519068    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b" podStartSLOduration=0.51904907 podStartE2EDuration="519.04907ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:00.518417905 +0000 UTC m=+3049.514265883" watchObservedRunningTime="2026-07-27 20:15:00.51904907 +0000 UTC m=+3049.514897057"
Jul 27 20:15:01.508656 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:01.508557    2568 generic.go:356] "Generic (PLEG): container finished" podID="9f108ba5-c3c7-458d-ad2a-98af57b83616" containerID="5c271c360f850ff79398d0b84ffbd9dbcbe36ae56d344bfba06a6cbaca057614" exitCode=0
Jul 27 20:15:01.508656 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:01.508636    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b" event={"ID":"9f108ba5-c3c7-458d-ad2a-98af57b83616","Type":"ContainerDied","Data":"5c271c360f850ff79398d0b84ffbd9dbcbe36ae56d344bfba06a6cbaca057614"}
Jul 27 20:15:01.816140 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:01.816107    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:01.878309 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:01.878274    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:02.104822 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:02.104676    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:02.319017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:02.318972    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:02.441487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:02.441402    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:02.540096 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:02.540055    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:02.646146 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:02.646122    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"
Jul 27 20:15:02.670370 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:02.670341    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:03.518783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:03.518761    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b"
Jul 27 20:15:03.518979 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:03.518758    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753055-f864b" event={"ID":"9f108ba5-c3c7-458d-ad2a-98af57b83616","Type":"ContainerDied","Data":"6ed0b4cd002adb7d0a82d96471d7686a8104e9976274b5f4e9989b3e4056b876"}
Jul 27 20:15:03.518979 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:03.518888    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed0b4cd002adb7d0a82d96471d7686a8104e9976274b5f4e9989b3e4056b876"
Jul 27 20:15:08.890696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:08.890665    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:08.950353 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:08.950326    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:09.018482 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:09.018449    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:09.121408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:09.121359    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:09.244656 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:09.244577    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:11.877084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:11.877060    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:15.478248 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:15.478210    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:15.578680 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:15.578648    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:22.195176 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:22.195142    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:22.488931 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:22.488847    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:22.725797 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:22.725760    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:25.534143 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.534106    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:25.613585 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.613548    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"]
Jul 27 20:15:25.613893 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.613881    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f108ba5-c3c7-458d-ad2a-98af57b83616" containerName="cleanup"
Jul 27 20:15:25.613945 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.613895    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f108ba5-c3c7-458d-ad2a-98af57b83616" containerName="cleanup"
Jul 27 20:15:25.613982 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.613969    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f108ba5-c3c7-458d-ad2a-98af57b83616" containerName="cleanup"
Jul 27 20:15:25.615497 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.615437    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:25.617261 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.617239    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.622646 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.622625    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1995ea52-serving-cert\""
Jul 27 20:15:25.632789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.632762    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"]
Jul 27 20:15:25.678896 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.678862    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:25.694905 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.694876    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrdg\" (UniqueName: \"kubernetes.io/projected/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-kube-api-access-5mrdg\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.695067 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.694947    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-maas-api-tls\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.695067 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.694979    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-openshift-service-ca\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.796132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.796038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrdg\" (UniqueName: \"kubernetes.io/projected/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-kube-api-access-5mrdg\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.796132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.796118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-maas-api-tls\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.796356 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.796155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-openshift-service-ca\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.796855 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.796831    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-openshift-service-ca\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.798719 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.798698    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-maas-api-tls\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.808194 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.808170    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrdg\" (UniqueName: \"kubernetes.io/projected/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-kube-api-access-5mrdg\") pod \"maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:25.837489 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.831872    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:25.930454 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:25.930412    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:26.069504 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.069481    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"]
Jul 27 20:15:26.071704 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:15:26.071670    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcd6ecf_90a4_4dc9_b363_661e4b795ec9.slice/crio-a76e9313fe9ee8e609a2db950177a1e851334cb7ed1dc6a1a118595a9e67e5ca WatchSource:0}: Error finding container a76e9313fe9ee8e609a2db950177a1e851334cb7ed1dc6a1a118595a9e67e5ca: Status 404 returned error can't find the container with id a76e9313fe9ee8e609a2db950177a1e851334cb7ed1dc6a1a118595a9e67e5ca
Jul 27 20:15:26.319161 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.319066    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:26.410588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.410556    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:26.411189 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.411171    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"]
Jul 27 20:15:26.414292 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.414278    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.417084 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.417064    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1995ea52-dockercfg-7rpf5\""
Jul 27 20:15:26.417231 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.417104    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1995ea52\""
Jul 27 20:15:26.424883 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.424854    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"]
Jul 27 20:15:26.434789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.434767    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"]
Jul 27 20:15:26.439196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.439176    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.452677 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.452652    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"]
Jul 27 20:15:26.502197 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.502137    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phczj\" (UniqueName: \"kubernetes.io/projected/47c71501-37eb-4d9c-830d-d4e548d97446-kube-api-access-phczj\") pod \"payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") " pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.502197 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.502187    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c71501-37eb-4d9c-830d-d4e548d97446-plugins-config-volume\") pod \"payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") " pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.502380 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.502210    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.502380 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.502255    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngzk\" (UniqueName: \"kubernetes.io/projected/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-kube-api-access-fngzk\") pod \"payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.603606 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.603493    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phczj\" (UniqueName: \"kubernetes.io/projected/47c71501-37eb-4d9c-830d-d4e548d97446-kube-api-access-phczj\") pod \"payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") " pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.603606 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.603563    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c71501-37eb-4d9c-830d-d4e548d97446-plugins-config-volume\") pod \"payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") " pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.603606 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.603586    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.604087 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.603619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngzk\" (UniqueName: \"kubernetes.io/projected/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-kube-api-access-fngzk\") pod \"payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.604308 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.604291    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c71501-37eb-4d9c-830d-d4e548d97446-plugins-config-volume\") pod \"payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") " pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.604349 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.604338    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.613176 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.613152    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phczj\" (UniqueName: \"kubernetes.io/projected/47c71501-37eb-4d9c-830d-d4e548d97446-kube-api-access-phczj\") pod \"payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") " pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.613803 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.613787    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngzk\" (UniqueName: \"kubernetes.io/projected/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-kube-api-access-fngzk\") pod \"payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.634904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.634872    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4" event={"ID":"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9","Type":"ContainerStarted","Data":"0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa"}
Jul 27 20:15:26.635026 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.634929    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4" event={"ID":"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9","Type":"ContainerStarted","Data":"a76e9313fe9ee8e609a2db950177a1e851334cb7ed1dc6a1a118595a9e67e5ca"}
Jul 27 20:15:26.635080 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.635037    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:26.658467 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.658416    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4" podStartSLOduration=1.300964682 podStartE2EDuration="1.65839778s" totalImagesPullingTime="357.433098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:26.656346922 +0000 UTC m=+3075.652194900" watchObservedRunningTime="2026-07-27 20:15:26.65839778 +0000 UTC m=+3075.654245757"
Jul 27 20:15:26.727201 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.727165    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:26.751211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.751182    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:26.882819 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.882775    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"]
Jul 27 20:15:26.885326 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:15:26.885288    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf25588_a6b7_4dd5_8e90_bf06aa0120b5.slice/crio-0923c66e46fb8772f384a11d62d26e6954dc21f76f1e3751ef7611694f1dc7ef WatchSource:0}: Error finding container 0923c66e46fb8772f384a11d62d26e6954dc21f76f1e3751ef7611694f1dc7ef: Status 404 returned error can't find the container with id 0923c66e46fb8772f384a11d62d26e6954dc21f76f1e3751ef7611694f1dc7ef
Jul 27 20:15:26.903508 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:26.903482    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"]
Jul 27 20:15:26.904557 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:15:26.904511    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c71501_37eb_4d9c_830d_d4e548d97446.slice/crio-a8ef6461d73f45f34881897f689581eff5653697331a945fbdf6172cb5ded29a WatchSource:0}: Error finding container a8ef6461d73f45f34881897f689581eff5653697331a945fbdf6172cb5ded29a: Status 404 returned error can't find the container with id a8ef6461d73f45f34881897f689581eff5653697331a945fbdf6172cb5ded29a
Jul 27 20:15:27.641233 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.641193    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj" event={"ID":"47c71501-37eb-4d9c-830d-d4e548d97446","Type":"ContainerStarted","Data":"a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d"}
Jul 27 20:15:27.641706 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.641247    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj" event={"ID":"47c71501-37eb-4d9c-830d-d4e548d97446","Type":"ContainerStarted","Data":"a8ef6461d73f45f34881897f689581eff5653697331a945fbdf6172cb5ded29a"}
Jul 27 20:15:27.641706 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.641343    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:27.642753 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.642730    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h" event={"ID":"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5","Type":"ContainerStarted","Data":"33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c"}
Jul 27 20:15:27.642865 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.642781    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h" event={"ID":"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5","Type":"ContainerStarted","Data":"0923c66e46fb8772f384a11d62d26e6954dc21f76f1e3751ef7611694f1dc7ef"}
Jul 27 20:15:27.661245 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.661195    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj" podStartSLOduration=1.661181963 podStartE2EDuration="1.661181963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:27.659134098 +0000 UTC m=+3076.654982077" watchObservedRunningTime="2026-07-27 20:15:27.661181963 +0000 UTC m=+3076.657029940"
Jul 27 20:15:27.677937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:27.677887    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h" podStartSLOduration=1.677867899 podStartE2EDuration="1.677867899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:27.677055325 +0000 UTC m=+3076.672903298" watchObservedRunningTime="2026-07-27 20:15:27.677867899 +0000 UTC m=+3076.673715902"
Jul 27 20:15:28.647567 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:28.647519    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:30.013660 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:30.013630    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:32.648960 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:32.648932    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:15:38.649872 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:38.649839    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:15:39.654277 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:39.654248    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:15:44.662721 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.662691    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"]
Jul 27 20:15:44.667140 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.667116    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.670013 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.669983    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1995ea52\"/\"e2e-lifecycle-model-1995ea52-kserve-self-signed-certs\""
Jul 27 20:15:44.671213 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.671190    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1995ea52\"/\"kube-root-ca.crt\""
Jul 27 20:15:44.671449 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.671432    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1995ea52\"/\"default-dockercfg-l2m4g\""
Jul 27 20:15:44.671513 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.671490    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1995ea52\"/\"openshift-service-ca.crt\""
Jul 27 20:15:44.677460 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.677438    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"]
Jul 27 20:15:44.694158 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.694129    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:15:44.761920 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.761892    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-model-cache\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.761920 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.761924    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-dshm\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.762170 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.761963    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tmp-dir\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.762170 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.762066    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tls-certs\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.762170 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.762101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-home\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.762170 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.762123    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7hq\" (UniqueName: \"kubernetes.io/projected/e8881db4-6d9e-471e-97c3-ba07a75da3c6-kube-api-access-lf7hq\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.863548 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.863470    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-dshm\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.863739 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.863576    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tmp-dir\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.863739 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.863634    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tls-certs\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.863739 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.863656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-home\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.863739 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.863679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7hq\" (UniqueName: \"kubernetes.io/projected/e8881db4-6d9e-471e-97c3-ba07a75da3c6-kube-api-access-lf7hq\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.863994 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.863773    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-model-cache\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.864058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.864016    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tmp-dir\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.864058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.864046    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-home\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.864158 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.864111    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-model-cache\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.865932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.865911    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-dshm\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.866172 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.866155    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tls-certs\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.874485 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.874462    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7hq\" (UniqueName: \"kubernetes.io/projected/e8881db4-6d9e-471e-97c3-ba07a75da3c6-kube-api-access-lf7hq\") pod \"e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") " pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:44.977701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:44.977615    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:45.112715 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:45.112471    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"]
Jul 27 20:15:45.114883 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:15:45.114856    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8881db4_6d9e_471e_97c3_ba07a75da3c6.slice/crio-1acdde3b5247f97d5c09af2fdf11a974e7592bd0ece7f39fef54f8d6dfe0a5ab WatchSource:0}: Error finding container 1acdde3b5247f97d5c09af2fdf11a974e7592bd0ece7f39fef54f8d6dfe0a5ab: Status 404 returned error can't find the container with id 1acdde3b5247f97d5c09af2fdf11a974e7592bd0ece7f39fef54f8d6dfe0a5ab
Jul 27 20:15:45.734850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:45.734811    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8" event={"ID":"e8881db4-6d9e-471e-97c3-ba07a75da3c6","Type":"ContainerStarted","Data":"cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7"}
Jul 27 20:15:45.735239 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:45.734858    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8" event={"ID":"e8881db4-6d9e-471e-97c3-ba07a75da3c6","Type":"ContainerStarted","Data":"1acdde3b5247f97d5c09af2fdf11a974e7592bd0ece7f39fef54f8d6dfe0a5ab"}
Jul 27 20:15:45.756361 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:45.756312    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8" podStartSLOduration=1.756297336 podStartE2EDuration="1.756297336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:15:45.755049251 +0000 UTC m=+3094.750897242" watchObservedRunningTime="2026-07-27 20:15:45.756297336 +0000 UTC m=+3094.752145318"
Jul 27 20:15:54.978712 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:54.978672    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:54.979221 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:54.978728    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:54.991061 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:54.991039    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:15:55.795598 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:15:55.795573    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:16:04.281771 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.281733    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"]
Jul 27 20:16:04.285314 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.285298    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:04.288234 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.288217    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:16:04.294050 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.294028    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"]
Jul 27 20:16:04.344500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.344463    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862a55ec-c383-419f-b94e-1d5eccd5200c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1995ea52-gscpg\" (UID: \"862a55ec-c383-419f-b94e-1d5eccd5200c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:04.445852 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.445811    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862a55ec-c383-419f-b94e-1d5eccd5200c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1995ea52-gscpg\" (UID: \"862a55ec-c383-419f-b94e-1d5eccd5200c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:04.446468 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.446436    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862a55ec-c383-419f-b94e-1d5eccd5200c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1995ea52-gscpg\" (UID: \"862a55ec-c383-419f-b94e-1d5eccd5200c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:04.597467 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.597364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:04.736282 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.736246    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"]
Jul 27 20:16:04.737197 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:16:04.737174    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862a55ec_c383_419f_b94e_1d5eccd5200c.slice/crio-f25efb8ea9a2fd4637a062bc83051a2a3f7ba8a167030578543979f3939e9598 WatchSource:0}: Error finding container f25efb8ea9a2fd4637a062bc83051a2a3f7ba8a167030578543979f3939e9598: Status 404 returned error can't find the container with id f25efb8ea9a2fd4637a062bc83051a2a3f7ba8a167030578543979f3939e9598
Jul 27 20:16:04.829258 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:04.829227    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg" event={"ID":"862a55ec-c383-419f-b94e-1d5eccd5200c","Type":"ContainerStarted","Data":"f25efb8ea9a2fd4637a062bc83051a2a3f7ba8a167030578543979f3939e9598"}
Jul 27 20:16:05.836085 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:05.836044    2568 generic.go:356] "Generic (PLEG): container finished" podID="862a55ec-c383-419f-b94e-1d5eccd5200c" containerID="6f4a3658e754ba53881ec7ef6e2fa3c88c45bb08c3377f8b3cb760e7a241a696" exitCode=0
Jul 27 20:16:05.836555 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:05.836128    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg" event={"ID":"862a55ec-c383-419f-b94e-1d5eccd5200c","Type":"ContainerDied","Data":"6f4a3658e754ba53881ec7ef6e2fa3c88c45bb08c3377f8b3cb760e7a241a696"}
Jul 27 20:16:06.975120 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:06.975097    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:07.071373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:07.071335    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862a55ec-c383-419f-b94e-1d5eccd5200c-maas-api-service-ca\") pod \"862a55ec-c383-419f-b94e-1d5eccd5200c\" (UID: \"862a55ec-c383-419f-b94e-1d5eccd5200c\") "
Jul 27 20:16:07.071716 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:07.071694    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862a55ec-c383-419f-b94e-1d5eccd5200c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "862a55ec-c383-419f-b94e-1d5eccd5200c" (UID: "862a55ec-c383-419f-b94e-1d5eccd5200c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:07.172549 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:07.172438    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/862a55ec-c383-419f-b94e-1d5eccd5200c-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:07.847088 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:07.847002    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg" event={"ID":"862a55ec-c383-419f-b94e-1d5eccd5200c","Type":"ContainerDied","Data":"f25efb8ea9a2fd4637a062bc83051a2a3f7ba8a167030578543979f3939e9598"}
Jul 27 20:16:07.847088 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:07.847048    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f25efb8ea9a2fd4637a062bc83051a2a3f7ba8a167030578543979f3939e9598"
Jul 27 20:16:07.847088 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:07.847012    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"
Jul 27 20:16:19.850551 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.850505    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"]
Jul 27 20:16:19.851055 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.850747    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4" podUID="ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" containerName="maas-api" containerID="cri-o://0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa" gracePeriod=30
Jul 27 20:16:19.872366 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.872340    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:16:19.907811 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.907777    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"]
Jul 27 20:16:19.908293 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.908265    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj" podUID="47c71501-37eb-4d9c-830d-d4e548d97446" containerName="payload-processing" containerID="cri-o://a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d" gracePeriod=30
Jul 27 20:16:19.910855 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.910829    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"]
Jul 27 20:16:19.911223 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.911193    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h" podUID="ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" containerName="payload-pre-processing" containerID="cri-o://33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c" gracePeriod=30
Jul 27 20:16:19.932425 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.932372    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:16:19.993604 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:19.993556    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:16:20.181438 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.181413    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:16:20.203160 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.203132    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:16:20.207344 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.207316    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:16:20.286764 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.286732    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c71501-37eb-4d9c-830d-d4e548d97446-plugins-config-volume\") pod \"47c71501-37eb-4d9c-830d-d4e548d97446\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") "
Jul 27 20:16:20.286953 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.286776    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-openshift-service-ca\") pod \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") "
Jul 27 20:16:20.286953 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.286819    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mrdg\" (UniqueName: \"kubernetes.io/projected/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-kube-api-access-5mrdg\") pod \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") "
Jul 27 20:16:20.286953 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.286840    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fngzk\" (UniqueName: \"kubernetes.io/projected/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-kube-api-access-fngzk\") pod \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") "
Jul 27 20:16:20.286953 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.286879    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phczj\" (UniqueName: \"kubernetes.io/projected/47c71501-37eb-4d9c-830d-d4e548d97446-kube-api-access-phczj\") pod \"47c71501-37eb-4d9c-830d-d4e548d97446\" (UID: \"47c71501-37eb-4d9c-830d-d4e548d97446\") "
Jul 27 20:16:20.286953 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.286908    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-plugins-config-volume\") pod \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\" (UID: \"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5\") "
Jul 27 20:16:20.287233 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.287053    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-maas-api-tls\") pod \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\" (UID: \"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9\") "
Jul 27 20:16:20.287298 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.287240    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" (UID: "ebcd6ecf-90a4-4dc9-b363-661e4b795ec9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:20.287365 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.287298    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" (UID: "ddf25588-a6b7-4dd5-8e90-bf06aa0120b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:20.287574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.287512    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.287574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.287556    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.287892 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.287838    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c71501-37eb-4d9c-830d-d4e548d97446-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47c71501-37eb-4d9c-830d-d4e548d97446" (UID: "47c71501-37eb-4d9c-830d-d4e548d97446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:16:20.289289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.289264    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c71501-37eb-4d9c-830d-d4e548d97446-kube-api-access-phczj" (OuterVolumeSpecName: "kube-api-access-phczj") pod "47c71501-37eb-4d9c-830d-d4e548d97446" (UID: "47c71501-37eb-4d9c-830d-d4e548d97446"). InnerVolumeSpecName "kube-api-access-phczj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:20.289289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.289275    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-kube-api-access-fngzk" (OuterVolumeSpecName: "kube-api-access-fngzk") pod "ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" (UID: "ddf25588-a6b7-4dd5-8e90-bf06aa0120b5"). InnerVolumeSpecName "kube-api-access-fngzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:20.289767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.289748    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" (UID: "ebcd6ecf-90a4-4dc9-b363-661e4b795ec9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:16:20.289767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.289750    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-kube-api-access-5mrdg" (OuterVolumeSpecName: "kube-api-access-5mrdg") pod "ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" (UID: "ebcd6ecf-90a4-4dc9-b363-661e4b795ec9"). InnerVolumeSpecName "kube-api-access-5mrdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:20.388308 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.388218    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.388308 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.388248    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c71501-37eb-4d9c-830d-d4e548d97446-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.388308 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.388258    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrdg\" (UniqueName: \"kubernetes.io/projected/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9-kube-api-access-5mrdg\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.388308 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.388267    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngzk\" (UniqueName: \"kubernetes.io/projected/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5-kube-api-access-fngzk\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.388308 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.388276    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phczj\" (UniqueName: \"kubernetes.io/projected/47c71501-37eb-4d9c-830d-d4e548d97446-kube-api-access-phczj\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:20.910953 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.910920    2568 generic.go:356] "Generic (PLEG): container finished" podID="ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" containerID="33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c" exitCode=0
Jul 27 20:16:20.911393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.910970    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"
Jul 27 20:16:20.911393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.911016    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h" event={"ID":"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5","Type":"ContainerDied","Data":"33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c"}
Jul 27 20:16:20.911393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.911077    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h" event={"ID":"ddf25588-a6b7-4dd5-8e90-bf06aa0120b5","Type":"ContainerDied","Data":"0923c66e46fb8772f384a11d62d26e6954dc21f76f1e3751ef7611694f1dc7ef"}
Jul 27 20:16:20.911393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.911105    2568 scope.go:117] "RemoveContainer" containerID="33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c"
Jul 27 20:16:20.912584 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.912361    2568 generic.go:356] "Generic (PLEG): container finished" podID="ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" containerID="0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa" exitCode=0
Jul 27 20:16:20.912584 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.912415    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"
Jul 27 20:16:20.912584 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.912459    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4" event={"ID":"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9","Type":"ContainerDied","Data":"0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa"}
Jul 27 20:16:20.912584 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.912513    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4" event={"ID":"ebcd6ecf-90a4-4dc9-b363-661e4b795ec9","Type":"ContainerDied","Data":"a76e9313fe9ee8e609a2db950177a1e851334cb7ed1dc6a1a118595a9e67e5ca"}
Jul 27 20:16:20.914088 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.914063    2568 generic.go:356] "Generic (PLEG): container finished" podID="47c71501-37eb-4d9c-830d-d4e548d97446" containerID="a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d" exitCode=0
Jul 27 20:16:20.914204 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.914111    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"
Jul 27 20:16:20.914204 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.914129    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj" event={"ID":"47c71501-37eb-4d9c-830d-d4e548d97446","Type":"ContainerDied","Data":"a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d"}
Jul 27 20:16:20.914204 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.914190    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj" event={"ID":"47c71501-37eb-4d9c-830d-d4e548d97446","Type":"ContainerDied","Data":"a8ef6461d73f45f34881897f689581eff5653697331a945fbdf6172cb5ded29a"}
Jul 27 20:16:20.931440 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.931417    2568 scope.go:117] "RemoveContainer" containerID="33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c"
Jul 27 20:16:20.931867 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:16:20.931814    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c\": container with ID starting with 33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c not found: ID does not exist" containerID="33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c"
Jul 27 20:16:20.932042 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.932005    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c"} err="failed to get container status \"33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c\": rpc error: code = NotFound desc = could not find container \"33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c\": container with ID starting with 33cf0f985418188935f82a90f1105af59c390d239de01ae4a4c4cc0aaf96ce3c not found: ID does not exist"
Jul 27 20:16:20.932154 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.932134    2568 scope.go:117] "RemoveContainer" containerID="0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa"
Jul 27 20:16:20.940991 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.940969    2568 scope.go:117] "RemoveContainer" containerID="0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa"
Jul 27 20:16:20.941285 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:16:20.941266    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa\": container with ID starting with 0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa not found: ID does not exist" containerID="0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa"
Jul 27 20:16:20.941351 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.941295    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa"} err="failed to get container status \"0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa\": rpc error: code = NotFound desc = could not find container \"0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa\": container with ID starting with 0241e4412784d203d3fe982cff1488b0339a6feb7b638f28fa6d1c46cba134aa not found: ID does not exist"
Jul 27 20:16:20.941351 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.941313    2568 scope.go:117] "RemoveContainer" containerID="a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d"
Jul 27 20:16:20.948540 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.948504    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"]
Jul 27 20:16:20.949666 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.949646    2568 scope.go:117] "RemoveContainer" containerID="a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d"
Jul 27 20:16:20.949940 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:16:20.949922    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d\": container with ID starting with a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d not found: ID does not exist" containerID="a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d"
Jul 27 20:16:20.949986 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.949948    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d"} err="failed to get container status \"a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d\": rpc error: code = NotFound desc = could not find container \"a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d\": container with ID starting with a2f7aa555bfeb97d241105d7a08dca9b663fe8cf049c9e5061e9626e5be0ed4d not found: ID does not exist"
Jul 27 20:16:20.953221 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.953197    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1995ea52-7b69dbdc6d-4b28h"]
Jul 27 20:16:20.966019 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.965994    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"]
Jul 27 20:16:20.969903 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.969879    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1995ea52-74bcc6bfc4-rw4jj"]
Jul 27 20:16:20.981918 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.981890    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"]
Jul 27 20:16:20.986704 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:20.986679    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1995ea52-8485747bbb-hq4b4"]
Jul 27 20:16:21.567926 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:21.567894    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c71501-37eb-4d9c-830d-d4e548d97446" path="/var/lib/kubelet/pods/47c71501-37eb-4d9c-830d-d4e548d97446/volumes"
Jul 27 20:16:21.568251 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:21.568239    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" path="/var/lib/kubelet/pods/ddf25588-a6b7-4dd5-8e90-bf06aa0120b5/volumes"
Jul 27 20:16:21.568582 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:21.568570    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" path="/var/lib/kubelet/pods/ebcd6ecf-90a4-4dc9-b363-661e4b795ec9/volumes"
Jul 27 20:16:24.369638 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:24.369602    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"]
Jul 27 20:16:24.370123 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:24.369722    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:16:24.371889 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:24.371868    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1995ea52-gscpg"]
Jul 27 20:16:25.567068 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:25.567034    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862a55ec-c383-419f-b94e-1d5eccd5200c" path="/var/lib/kubelet/pods/862a55ec-c383-419f-b94e-1d5eccd5200c/volumes"
Jul 27 20:16:27.520994 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:27.520954    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:16:32.510080 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:32.510001    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"]
Jul 27 20:16:32.561428 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:32.561396    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:16:47.529863 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.529793    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8" podUID="e8881db4-6d9e-471e-97c3-ba07a75da3c6" containerName="main" containerID="cri-o://cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7" gracePeriod=45
Jul 27 20:16:47.774884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.774857    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:16:47.942635 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942501    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-home\") pod \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") "
Jul 27 20:16:47.942635 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942585    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-model-cache\") pod \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") "
Jul 27 20:16:47.942635 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942603    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tmp-dir\") pod \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") "
Jul 27 20:16:47.942939 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942654    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf7hq\" (UniqueName: \"kubernetes.io/projected/e8881db4-6d9e-471e-97c3-ba07a75da3c6-kube-api-access-lf7hq\") pod \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") "
Jul 27 20:16:47.942939 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942808    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-dshm\") pod \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") "
Jul 27 20:16:47.942939 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942852    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tls-certs\") pod \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\" (UID: \"e8881db4-6d9e-471e-97c3-ba07a75da3c6\") "
Jul 27 20:16:47.942939 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942844    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8881db4-6d9e-471e-97c3-ba07a75da3c6" (UID: "e8881db4-6d9e-471e-97c3-ba07a75da3c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:47.942939 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942901    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8881db4-6d9e-471e-97c3-ba07a75da3c6" (UID: "e8881db4-6d9e-471e-97c3-ba07a75da3c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:47.942939 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.942861    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-home" (OuterVolumeSpecName: "home") pod "e8881db4-6d9e-471e-97c3-ba07a75da3c6" (UID: "e8881db4-6d9e-471e-97c3-ba07a75da3c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:47.943200 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.943112    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:47.943200 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.943128    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:47.943200 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.943140    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:47.945149 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.945119    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8881db4-6d9e-471e-97c3-ba07a75da3c6-kube-api-access-lf7hq" (OuterVolumeSpecName: "kube-api-access-lf7hq") pod "e8881db4-6d9e-471e-97c3-ba07a75da3c6" (UID: "e8881db4-6d9e-471e-97c3-ba07a75da3c6"). InnerVolumeSpecName "kube-api-access-lf7hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:16:47.945149 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.945127    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8881db4-6d9e-471e-97c3-ba07a75da3c6" (UID: "e8881db4-6d9e-471e-97c3-ba07a75da3c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:16:47.945323 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:47.945122    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-dshm" (OuterVolumeSpecName: "dshm") pod "e8881db4-6d9e-471e-97c3-ba07a75da3c6" (UID: "e8881db4-6d9e-471e-97c3-ba07a75da3c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:16:48.044005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.043972    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8881db4-6d9e-471e-97c3-ba07a75da3c6-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:48.044005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.044007    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8881db4-6d9e-471e-97c3-ba07a75da3c6-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:48.044215 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.044023    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7hq\" (UniqueName: \"kubernetes.io/projected/e8881db4-6d9e-471e-97c3-ba07a75da3c6-kube-api-access-lf7hq\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:16:48.047282 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.047251    2568 generic.go:356] "Generic (PLEG): container finished" podID="e8881db4-6d9e-471e-97c3-ba07a75da3c6" containerID="cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7" exitCode=0
Jul 27 20:16:48.047414 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.047296    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"
Jul 27 20:16:48.047414 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.047341    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8" event={"ID":"e8881db4-6d9e-471e-97c3-ba07a75da3c6","Type":"ContainerDied","Data":"cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7"}
Jul 27 20:16:48.047414 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.047391    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8" event={"ID":"e8881db4-6d9e-471e-97c3-ba07a75da3c6","Type":"ContainerDied","Data":"1acdde3b5247f97d5c09af2fdf11a974e7592bd0ece7f39fef54f8d6dfe0a5ab"}
Jul 27 20:16:48.047515 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.047415    2568 scope.go:117] "RemoveContainer" containerID="cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7"
Jul 27 20:16:48.056850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.056828    2568 scope.go:117] "RemoveContainer" containerID="cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7"
Jul 27 20:16:48.057132 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:16:48.057114    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7\": container with ID starting with cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7 not found: ID does not exist" containerID="cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7"
Jul 27 20:16:48.057203 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.057141    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7"} err="failed to get container status \"cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7\": rpc error: code = NotFound desc = could not find container \"cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7\": container with ID starting with cabe55fd5dc6e28aeaa674893a3f21a0f7ffaee5d83d2945266b9c5031a06fb7 not found: ID does not exist"
Jul 27 20:16:48.071758 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.071732    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"]
Jul 27 20:16:48.076201 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:48.076175    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1995ea52/e2e-lifecycle-model-1995ea52-kserve-7794569599-q2qh8"]
Jul 27 20:16:49.568077 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:16:49.568043    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8881db4-6d9e-471e-97c3-ba07a75da3c6" path="/var/lib/kubelet/pods/e8881db4-6d9e-471e-97c3-ba07a75da3c6/volumes"
Jul 27 20:17:08.740035 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:08.740001    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:10.512825 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:10.512794    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:16.831304 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:16.831246    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:16.932049 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:16.932017    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:17.119577 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:17.119462    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:17.338993 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:17.338954    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:18.400902 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:18.400871    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:18.500554 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:18.500491    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:18.600624 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:18.600595    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:19.275033 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:19.274996    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:19.370765 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:19.370735    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:19.474035 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:19.474003    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:20.804091 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:20.804057    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:27.222479 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:27.222443    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:27.280660 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:27.280623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:27.878956 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:27.878925    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:30.252554 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:30.252499    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:30.308696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:30.308650    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:34.850855 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:34.850765    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:34.951610 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:34.951578    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:35.051275 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:35.051234    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:45.574983 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:45.574949    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:46.054313 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:46.054273    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:17:50.790128 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790091    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4tktz"]
Jul 27 20:17:50.790619 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790598    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c71501-37eb-4d9c-830d-d4e548d97446" containerName="payload-processing"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790621    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c71501-37eb-4d9c-830d-d4e548d97446" containerName="payload-processing"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790648    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" containerName="maas-api"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790657    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" containerName="maas-api"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790670    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" containerName="payload-pre-processing"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790680    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" containerName="payload-pre-processing"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790690    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862a55ec-c383-419f-b94e-1d5eccd5200c" containerName="revoke-keys"
Jul 27 20:17:50.790701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790698    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="862a55ec-c383-419f-b94e-1d5eccd5200c" containerName="revoke-keys"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790715    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8881db4-6d9e-471e-97c3-ba07a75da3c6" containerName="main"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790724    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8881db4-6d9e-471e-97c3-ba07a75da3c6" containerName="main"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790821    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcd6ecf-90a4-4dc9-b363-661e4b795ec9" containerName="maas-api"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790835    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="862a55ec-c383-419f-b94e-1d5eccd5200c" containerName="revoke-keys"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790843    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf25588-a6b7-4dd5-8e90-bf06aa0120b5" containerName="payload-pre-processing"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790854    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8881db4-6d9e-471e-97c3-ba07a75da3c6" containerName="main"
Jul 27 20:17:50.791041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.790866    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c71501-37eb-4d9c-830d-d4e548d97446" containerName="payload-processing"
Jul 27 20:17:50.794014 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.793994    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:17:50.796715 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.796695    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sttwx\""
Jul 27 20:17:50.802847 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.802821    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4tktz"]
Jul 27 20:17:50.915088 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:50.915058    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8zg\" (UniqueName: \"kubernetes.io/projected/f292a5a4-27d6-49fc-85db-943353b70cb0-kube-api-access-zz8zg\") pod \"kuadrant-operator-catalog-4tktz\" (UID: \"f292a5a4-27d6-49fc-85db-943353b70cb0\") " pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:17:51.016538 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:51.016498    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8zg\" (UniqueName: \"kubernetes.io/projected/f292a5a4-27d6-49fc-85db-943353b70cb0-kube-api-access-zz8zg\") pod \"kuadrant-operator-catalog-4tktz\" (UID: \"f292a5a4-27d6-49fc-85db-943353b70cb0\") " pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:17:51.026101 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:51.026067    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8zg\" (UniqueName: \"kubernetes.io/projected/f292a5a4-27d6-49fc-85db-943353b70cb0-kube-api-access-zz8zg\") pod \"kuadrant-operator-catalog-4tktz\" (UID: \"f292a5a4-27d6-49fc-85db-943353b70cb0\") " pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:17:51.104055 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:51.103971    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:17:51.236102 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:51.236071    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4tktz"]
Jul 27 20:17:51.238614 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:17:51.238586    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf292a5a4_27d6_49fc_85db_943353b70cb0.slice/crio-af5062ef4e85ef1672df7239c33caf19b937e04714286bf039bb1b2d81d3195e WatchSource:0}: Error finding container af5062ef4e85ef1672df7239c33caf19b937e04714286bf039bb1b2d81d3195e: Status 404 returned error can't find the container with id af5062ef4e85ef1672df7239c33caf19b937e04714286bf039bb1b2d81d3195e
Jul 27 20:17:51.358935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:51.358858    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4tktz" event={"ID":"f292a5a4-27d6-49fc-85db-943353b70cb0","Type":"ContainerStarted","Data":"af5062ef4e85ef1672df7239c33caf19b937e04714286bf039bb1b2d81d3195e"}
Jul 27 20:17:54.379320 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:54.379279    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4tktz" event={"ID":"f292a5a4-27d6-49fc-85db-943353b70cb0","Type":"ContainerStarted","Data":"7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3"}
Jul 27 20:17:54.400709 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:17:54.400663    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4tktz" podStartSLOduration=2.244526404 podStartE2EDuration="4.400648374s" totalImagesPullingTime="2.15612197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:17:54.399207194 +0000 UTC m=+3223.395055193" watchObservedRunningTime="2026-07-27 20:17:54.400648374 +0000 UTC m=+3223.396496371"
Jul 27 20:18:01.104211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:01.104170    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:18:01.104629 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:01.104272    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:18:01.125761 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:01.125731    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:18:01.439172 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:01.439103    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:18:01.481945 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:01.481908    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4tktz"]
Jul 27 20:18:03.426359 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:03.426317    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4tktz" podUID="f292a5a4-27d6-49fc-85db-943353b70cb0" containerName="registry-server" containerID="cri-o://7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3" gracePeriod=2
Jul 27 20:18:03.673400 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:03.673375    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:18:03.736423 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:03.736346    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz8zg\" (UniqueName: \"kubernetes.io/projected/f292a5a4-27d6-49fc-85db-943353b70cb0-kube-api-access-zz8zg\") pod \"f292a5a4-27d6-49fc-85db-943353b70cb0\" (UID: \"f292a5a4-27d6-49fc-85db-943353b70cb0\") "
Jul 27 20:18:03.738462 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:03.738440    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f292a5a4-27d6-49fc-85db-943353b70cb0-kube-api-access-zz8zg" (OuterVolumeSpecName: "kube-api-access-zz8zg") pod "f292a5a4-27d6-49fc-85db-943353b70cb0" (UID: "f292a5a4-27d6-49fc-85db-943353b70cb0"). InnerVolumeSpecName "kube-api-access-zz8zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:18:03.837195 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:03.837164    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz8zg\" (UniqueName: \"kubernetes.io/projected/f292a5a4-27d6-49fc-85db-943353b70cb0-kube-api-access-zz8zg\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:18:04.431567 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.431534    2568 generic.go:356] "Generic (PLEG): container finished" podID="f292a5a4-27d6-49fc-85db-943353b70cb0" containerID="7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3" exitCode=0
Jul 27 20:18:04.432005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.431577    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4tktz"
Jul 27 20:18:04.432005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.431619    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4tktz" event={"ID":"f292a5a4-27d6-49fc-85db-943353b70cb0","Type":"ContainerDied","Data":"7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3"}
Jul 27 20:18:04.432005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.431669    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4tktz" event={"ID":"f292a5a4-27d6-49fc-85db-943353b70cb0","Type":"ContainerDied","Data":"af5062ef4e85ef1672df7239c33caf19b937e04714286bf039bb1b2d81d3195e"}
Jul 27 20:18:04.432005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.431689    2568 scope.go:117] "RemoveContainer" containerID="7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3"
Jul 27 20:18:04.441052 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.441035    2568 scope.go:117] "RemoveContainer" containerID="7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3"
Jul 27 20:18:04.441337 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:18:04.441320    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3\": container with ID starting with 7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3 not found: ID does not exist" containerID="7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3"
Jul 27 20:18:04.441389 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.441346    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3"} err="failed to get container status \"7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3\": rpc error: code = NotFound desc = could not find container \"7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3\": container with ID starting with 7d7357554153fa3b153c2a6d928eb04bbf5e1a947112b3c76fa323b5ffb37fa3 not found: ID does not exist"
Jul 27 20:18:04.455506 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.455482    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4tktz"]
Jul 27 20:18:04.462485 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:04.462463    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4tktz"]
Jul 27 20:18:05.566774 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:05.566740    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f292a5a4-27d6-49fc-85db-943353b70cb0" path="/var/lib/kubelet/pods/f292a5a4-27d6-49fc-85db-943353b70cb0/volumes"
Jul 27 20:18:06.882936 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:06.882903    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:14.033535 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:14.033497    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:14.137868 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:14.137811    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:14.400665 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:14.400573    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:14.711109 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:14.710994    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:14.806589 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:14.806558    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:17.469940 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.469904    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:17.550927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.550878    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:17.555540 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.555494    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"]
Jul 27 20:18:17.556041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.556023    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f292a5a4-27d6-49fc-85db-943353b70cb0" containerName="registry-server"
Jul 27 20:18:17.556041 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.556043    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f292a5a4-27d6-49fc-85db-943353b70cb0" containerName="registry-server"
Jul 27 20:18:17.556191 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.556163    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f292a5a4-27d6-49fc-85db-943353b70cb0" containerName="registry-server"
Jul 27 20:18:17.560754 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.560734    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.563333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.563312    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-71745f-serving-cert\""
Jul 27 20:18:17.563611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.563423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.563611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.563473    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2xc\" (UniqueName: \"kubernetes.io/projected/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-kube-api-access-js2xc\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.563781 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.563615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.570279 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.570253    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"]
Jul 27 20:18:17.627439 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.627394    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:17.664513 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.664477    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.664723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.664564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.664723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.664588    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2xc\" (UniqueName: \"kubernetes.io/projected/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-kube-api-access-js2xc\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.664723 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:18:17.664663    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-serving-cert: secret "maas-api-e2e-inf-a-71745f-serving-cert" not found
Jul 27 20:18:17.664841 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:18:17.664735    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls podName:ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b nodeName:}" failed. No retries permitted until 2026-07-27 20:18:18.164718495 +0000 UTC m=+3247.160566452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls") pod "maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" (UID: "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b") : secret "maas-api-e2e-inf-a-71745f-serving-cert" not found
Jul 27 20:18:17.665201 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.665184    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.674438 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.674413    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2xc\" (UniqueName: \"kubernetes.io/projected/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-kube-api-access-js2xc\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:17.890696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:17.890667    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:18.073148 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.073119    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:18.168160 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.168061    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:18.169800 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.169775    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"]
Jul 27 20:18:18.170842 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.170820    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls\") pod \"maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:18.171921 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.171896    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:18.173465 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.173445    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.174946 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.174930    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:18.176157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.176140    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-71745f\""
Jul 27 20:18:18.184811 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.184648    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"]
Jul 27 20:18:18.221776 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.221740    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"]
Jul 27 20:18:18.227069 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.227041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.230096 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.230067    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-71745f-dockercfg-d9ftv\""
Jul 27 20:18:18.236867 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.236797    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"]
Jul 27 20:18:18.321333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.321303    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"]
Jul 27 20:18:18.323376 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:18:18.323349    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4d8c9b_4922_4bed_b97b_5f00f26d0c0b.slice/crio-c08902324e242bb07243e7634166fd0ed0a14c2879285efd4c32b0b87b4c5d44 WatchSource:0}: Error finding container c08902324e242bb07243e7634166fd0ed0a14c2879285efd4c32b0b87b4c5d44: Status 404 returned error can't find the container with id c08902324e242bb07243e7634166fd0ed0a14c2879285efd4c32b0b87b4c5d44
Jul 27 20:18:18.370101 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.370063    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwp4\" (UniqueName: \"kubernetes.io/projected/eeeb4e80-d42d-4649-85ed-b08d02883175-kube-api-access-9pwp4\") pod \"payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.370272 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.370117    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeeb4e80-d42d-4649-85ed-b08d02883175-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.370272 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.370252    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac797eb-f5f0-47e8-980e-58de481eba0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.370351 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.370300    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8tn\" (UniqueName: \"kubernetes.io/projected/0ac797eb-f5f0-47e8-980e-58de481eba0a-kube-api-access-7b8tn\") pod \"payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.471017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.470938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac797eb-f5f0-47e8-980e-58de481eba0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.471017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.470977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8tn\" (UniqueName: \"kubernetes.io/projected/0ac797eb-f5f0-47e8-980e-58de481eba0a-kube-api-access-7b8tn\") pod \"payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.471017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.471019    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwp4\" (UniqueName: \"kubernetes.io/projected/eeeb4e80-d42d-4649-85ed-b08d02883175-kube-api-access-9pwp4\") pod \"payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.471473 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.471043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeeb4e80-d42d-4649-85ed-b08d02883175-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.471651 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.471632    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac797eb-f5f0-47e8-980e-58de481eba0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.471698 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.471678    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeeb4e80-d42d-4649-85ed-b08d02883175-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.480353 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.480321    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwp4\" (UniqueName: \"kubernetes.io/projected/eeeb4e80-d42d-4649-85ed-b08d02883175-kube-api-access-9pwp4\") pod \"payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.480552 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.480508    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8tn\" (UniqueName: \"kubernetes.io/projected/0ac797eb-f5f0-47e8-980e-58de481eba0a-kube-api-access-7b8tn\") pod \"payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.502989 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.502953    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" event={"ID":"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b","Type":"ContainerStarted","Data":"c08902324e242bb07243e7634166fd0ed0a14c2879285efd4c32b0b87b4c5d44"}
Jul 27 20:18:18.505136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.505118    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:18.546922 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.546870    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:18.649335 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.649298    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"]
Jul 27 20:18:18.651718 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:18:18.651685    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeeb4e80_d42d_4649_85ed_b08d02883175.slice/crio-a1e96be13564a253c446ca8e64ceee8d7a56c4ffdc009dc2b03b78755d0a62df WatchSource:0}: Error finding container a1e96be13564a253c446ca8e64ceee8d7a56c4ffdc009dc2b03b78755d0a62df: Status 404 returned error can't find the container with id a1e96be13564a253c446ca8e64ceee8d7a56c4ffdc009dc2b03b78755d0a62df
Jul 27 20:18:18.696298 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:18.696232    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"]
Jul 27 20:18:18.701651 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:18:18.701628    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac797eb_f5f0_47e8_980e_58de481eba0a.slice/crio-280d1e2e119ed1074a7e11ed25dca3610657946296b9b23ac929447c90ce0d0d WatchSource:0}: Error finding container 280d1e2e119ed1074a7e11ed25dca3610657946296b9b23ac929447c90ce0d0d: Status 404 returned error can't find the container with id 280d1e2e119ed1074a7e11ed25dca3610657946296b9b23ac929447c90ce0d0d
Jul 27 20:18:19.509882 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.509841    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" event={"ID":"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b","Type":"ContainerStarted","Data":"d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c"}
Jul 27 20:18:19.510352 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.509947    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:19.511498 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.511474    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc" event={"ID":"0ac797eb-f5f0-47e8-980e-58de481eba0a","Type":"ContainerStarted","Data":"e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775"}
Jul 27 20:18:19.511639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.511517    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc" event={"ID":"0ac797eb-f5f0-47e8-980e-58de481eba0a","Type":"ContainerStarted","Data":"280d1e2e119ed1074a7e11ed25dca3610657946296b9b23ac929447c90ce0d0d"}
Jul 27 20:18:19.511639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.511562    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:19.512788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.512762    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp" event={"ID":"eeeb4e80-d42d-4649-85ed-b08d02883175","Type":"ContainerStarted","Data":"397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19"}
Jul 27 20:18:19.512887 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.512809    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:19.512887 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.512829    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp" event={"ID":"eeeb4e80-d42d-4649-85ed-b08d02883175","Type":"ContainerStarted","Data":"a1e96be13564a253c446ca8e64ceee8d7a56c4ffdc009dc2b03b78755d0a62df"}
Jul 27 20:18:19.529575 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.529517    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" podStartSLOduration=2.073680085 podStartE2EDuration="2.529502913s" totalImagesPullingTime="455.822828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:19.5279441 +0000 UTC m=+3248.523792080" watchObservedRunningTime="2026-07-27 20:18:19.529502913 +0000 UTC m=+3248.525350891"
Jul 27 20:18:19.547005 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.546955    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp" podStartSLOduration=1.546942563 podStartE2EDuration="1.546942563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:19.545548431 +0000 UTC m=+3248.541396427" watchObservedRunningTime="2026-07-27 20:18:19.546942563 +0000 UTC m=+3248.542790541"
Jul 27 20:18:19.563236 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:19.563185    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc" podStartSLOduration=1.56317017 podStartE2EDuration="1.56317017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:19.562152759 +0000 UTC m=+3248.558000738" watchObservedRunningTime="2026-07-27 20:18:19.56317017 +0000 UTC m=+3248.559018147"
Jul 27 20:18:21.981425 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:21.981388    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:22.336609 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:22.336577    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:22.683281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:22.683190    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:22.986767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:22.986663    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:25.508038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.508003    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:25.522759 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.522735    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:18:25.609178 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.609141    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:25.616789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.616765    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"]
Jul 27 20:18:25.621807 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.621792    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.624643 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.624624    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-49ddd1-serving-cert\""
Jul 27 20:18:25.632193 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.632171    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"]
Jul 27 20:18:25.677664 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.677634    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:25.737450 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.737421    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.737650 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.737472    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bzz\" (UniqueName: \"kubernetes.io/projected/689236a4-31db-4a29-b4ff-5137948d0897-kube-api-access-v7bzz\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.737650 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.737495    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/689236a4-31db-4a29-b4ff-5137948d0897-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.838386 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.838353    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bzz\" (UniqueName: \"kubernetes.io/projected/689236a4-31db-4a29-b4ff-5137948d0897-kube-api-access-v7bzz\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.838573 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.838392    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/689236a4-31db-4a29-b4ff-5137948d0897-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.838573 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.838444    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.838660 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:18:25.838579    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-serving-cert: secret "maas-api-e2e-inf-b-49ddd1-serving-cert" not found
Jul 27 20:18:25.838660 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:18:25.838633    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls podName:689236a4-31db-4a29-b4ff-5137948d0897 nodeName:}" failed. No retries permitted until 2026-07-27 20:18:26.338617331 +0000 UTC m=+3255.334465287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls") pod "maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" (UID: "689236a4-31db-4a29-b4ff-5137948d0897") : secret "maas-api-e2e-inf-b-49ddd1-serving-cert" not found
Jul 27 20:18:25.839107 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.839089    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/689236a4-31db-4a29-b4ff-5137948d0897-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:25.849167 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:25.849143    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bzz\" (UniqueName: \"kubernetes.io/projected/689236a4-31db-4a29-b4ff-5137948d0897-kube-api-access-v7bzz\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:26.160964 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.160871    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:26.337092 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.337042    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:26.343255 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.343235    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:26.345840 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.345811    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls\") pod \"maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:26.437257 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.437165    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:26.465925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.465897    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"]
Jul 27 20:18:26.469948 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.469928    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"]
Jul 27 20:18:26.470067 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.470044    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.472931 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.472908    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-49ddd1\""
Jul 27 20:18:26.473273 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.473258    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.479234 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.479117    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"]
Jul 27 20:18:26.482861 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.482794    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"]
Jul 27 20:18:26.533064 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.533040    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:26.545574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.545551    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e827300-504b-4fbb-a43e-b4dddda30282-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.545721 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.545615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63326c4f-00ca-479e-92b3-a1313b49b0f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.545721 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.545652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59dlr\" (UniqueName: \"kubernetes.io/projected/63326c4f-00ca-479e-92b3-a1313b49b0f1-kube-api-access-59dlr\") pod \"payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.545721 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.545695    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krs4x\" (UniqueName: \"kubernetes.io/projected/4e827300-504b-4fbb-a43e-b4dddda30282-kube-api-access-krs4x\") pod \"payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.647012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.646982    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63326c4f-00ca-479e-92b3-a1313b49b0f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.647197 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.647046    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59dlr\" (UniqueName: \"kubernetes.io/projected/63326c4f-00ca-479e-92b3-a1313b49b0f1-kube-api-access-59dlr\") pod \"payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.647197 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.647141    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krs4x\" (UniqueName: \"kubernetes.io/projected/4e827300-504b-4fbb-a43e-b4dddda30282-kube-api-access-krs4x\") pod \"payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.647315 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.647204    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e827300-504b-4fbb-a43e-b4dddda30282-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.648026 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.647999    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63326c4f-00ca-479e-92b3-a1313b49b0f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.648160 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.648141    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e827300-504b-4fbb-a43e-b4dddda30282-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.657036 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.657012    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59dlr\" (UniqueName: \"kubernetes.io/projected/63326c4f-00ca-479e-92b3-a1313b49b0f1-kube-api-access-59dlr\") pod \"payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.657150 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.657119    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krs4x\" (UniqueName: \"kubernetes.io/projected/4e827300-504b-4fbb-a43e-b4dddda30282-kube-api-access-krs4x\") pod \"payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") " pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.673117 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.673094    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"]
Jul 27 20:18:26.675189 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:18:26.675166    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689236a4_31db_4a29_b4ff_5137948d0897.slice/crio-8ab588f3c93b62b8a83dd6b3d51bf8dfd3483cd8fdb6b170dd2a0adfafc36774 WatchSource:0}: Error finding container 8ab588f3c93b62b8a83dd6b3d51bf8dfd3483cd8fdb6b170dd2a0adfafc36774: Status 404 returned error can't find the container with id 8ab588f3c93b62b8a83dd6b3d51bf8dfd3483cd8fdb6b170dd2a0adfafc36774
Jul 27 20:18:26.784191 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.784166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:26.790554 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.790512    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:26.931892 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.931856    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"]
Jul 27 20:18:26.934358 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:18:26.934330    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63326c4f_00ca_479e_92b3_a1313b49b0f1.slice/crio-aa1e55c89c6cdb7806869b983bea2eba5e3b452cd323a07d23adaeaa63127e9f WatchSource:0}: Error finding container aa1e55c89c6cdb7806869b983bea2eba5e3b452cd323a07d23adaeaa63127e9f: Status 404 returned error can't find the container with id aa1e55c89c6cdb7806869b983bea2eba5e3b452cd323a07d23adaeaa63127e9f
Jul 27 20:18:26.954828 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:26.954758    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"]
Jul 27 20:18:26.955939 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:18:26.955911    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e827300_504b_4fbb_a43e_b4dddda30282.slice/crio-dfc95cb722a59b4cb9a97ecb83f38af5c9f1b841c5c0f4261d3298040286d6c8 WatchSource:0}: Error finding container dfc95cb722a59b4cb9a97ecb83f38af5c9f1b841c5c0f4261d3298040286d6c8: Status 404 returned error can't find the container with id dfc95cb722a59b4cb9a97ecb83f38af5c9f1b841c5c0f4261d3298040286d6c8
Jul 27 20:18:27.552972 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.552937    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9" event={"ID":"4e827300-504b-4fbb-a43e-b4dddda30282","Type":"ContainerStarted","Data":"2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f"}
Jul 27 20:18:27.553420 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.552991    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9" event={"ID":"4e827300-504b-4fbb-a43e-b4dddda30282","Type":"ContainerStarted","Data":"dfc95cb722a59b4cb9a97ecb83f38af5c9f1b841c5c0f4261d3298040286d6c8"}
Jul 27 20:18:27.553420 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.553054    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:27.554564 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.554510    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" event={"ID":"689236a4-31db-4a29-b4ff-5137948d0897","Type":"ContainerStarted","Data":"c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38"}
Jul 27 20:18:27.554636 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.554587    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:27.554636 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.554610    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" event={"ID":"689236a4-31db-4a29-b4ff-5137948d0897","Type":"ContainerStarted","Data":"8ab588f3c93b62b8a83dd6b3d51bf8dfd3483cd8fdb6b170dd2a0adfafc36774"}
Jul 27 20:18:27.556068 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.556045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" event={"ID":"63326c4f-00ca-479e-92b3-a1313b49b0f1","Type":"ContainerStarted","Data":"1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69"}
Jul 27 20:18:27.556192 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.556083    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" event={"ID":"63326c4f-00ca-479e-92b3-a1313b49b0f1","Type":"ContainerStarted","Data":"aa1e55c89c6cdb7806869b983bea2eba5e3b452cd323a07d23adaeaa63127e9f"}
Jul 27 20:18:27.556192 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.556140    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:18:27.571984 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.571934    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9" podStartSLOduration=1.571920482 podStartE2EDuration="1.571920482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:27.570779798 +0000 UTC m=+3256.566627776" watchObservedRunningTime="2026-07-27 20:18:27.571920482 +0000 UTC m=+3256.567768459"
Jul 27 20:18:27.587395 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.587344    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" podStartSLOduration=1.587326961 podStartE2EDuration="1.587326961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:27.586936021 +0000 UTC m=+3256.582783996" watchObservedRunningTime="2026-07-27 20:18:27.587326961 +0000 UTC m=+3256.583174920"
Jul 27 20:18:27.604861 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.604812    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" podStartSLOduration=2.292643401 podStartE2EDuration="2.604800213s" totalImagesPullingTime="312.156812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:18:27.603841392 +0000 UTC m=+3256.599689396" watchObservedRunningTime="2026-07-27 20:18:27.604800213 +0000 UTC m=+3256.600648191"
Jul 27 20:18:27.936012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:27.935932    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:30.034330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:30.034298    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:30.519410 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:30.519377    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:18:30.519661 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:30.519456    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:18:33.493238 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:33.493193    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:18:33.568733 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:33.568711    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:18:38.564169 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:38.564106    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" podUID="63326c4f-00ca-479e-92b3-a1313b49b0f1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.78:9004: connect: connection refused"
Jul 27 20:18:38.564649 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:38.564366    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:18:48.562935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:18:48.562904    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:19:06.138025 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.137993    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"]
Jul 27 20:19:06.142788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.142765    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.145722 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.145699    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49ddd1\"/\"kube-root-ca.crt\""
Jul 27 20:19:06.146991 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.146968    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49ddd1\"/\"default-dockercfg-mvtqk\""
Jul 27 20:19:06.147153 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.147129    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-49ddd1\"/\"openshift-service-ca.crt\""
Jul 27 20:19:06.147276 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.147199    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-49ddd1\"/\"test-model-49ddd1-kserve-self-signed-certs\""
Jul 27 20:19:06.153871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.153847    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"]
Jul 27 20:19:06.174574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.174517    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:19:06.211166 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.211134    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9f8\" (UniqueName: \"kubernetes.io/projected/c505919a-ad40-426e-9a96-b30eedd89231-kube-api-access-9g9f8\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.211166 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.211168    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-model-cache\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.211386 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.211267    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-dshm\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.211386 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.211307    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c505919a-ad40-426e-9a96-b30eedd89231-tls-certs\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.211386 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.211329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-home\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.211386 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.211367    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-tmp-dir\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.312510 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312471    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c505919a-ad40-426e-9a96-b30eedd89231-tls-certs\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.312712 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312513    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-home\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.312712 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-tmp-dir\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.312785 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312748    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9f8\" (UniqueName: \"kubernetes.io/projected/c505919a-ad40-426e-9a96-b30eedd89231-kube-api-access-9g9f8\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.312832 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312798    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-model-cache\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.312925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312911    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-dshm\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.313002 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.312985    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-tmp-dir\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.313063 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.313008    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-home\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.313247 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.313227    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-model-cache\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.315107 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.315084    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-dshm\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.315191 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.315172    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c505919a-ad40-426e-9a96-b30eedd89231-tls-certs\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.323743 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.323722    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9f8\" (UniqueName: \"kubernetes.io/projected/c505919a-ad40-426e-9a96-b30eedd89231-kube-api-access-9g9f8\") pod \"test-model-49ddd1-kserve-85b78f6f65-b7fzn\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") " pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.455379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.455295    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:06.595315 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.595293    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"]
Jul 27 20:19:06.597440 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:19:06.597409    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc505919a_ad40_426e_9a96_b30eedd89231.slice/crio-06a6e99dab8e686472cf00f7372c476d99d4778ff0d371af20714d2a5c28ecf1 WatchSource:0}: Error finding container 06a6e99dab8e686472cf00f7372c476d99d4778ff0d371af20714d2a5c28ecf1: Status 404 returned error can't find the container with id 06a6e99dab8e686472cf00f7372c476d99d4778ff0d371af20714d2a5c28ecf1
Jul 27 20:19:06.751595 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.751481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn" event={"ID":"c505919a-ad40-426e-9a96-b30eedd89231","Type":"ContainerStarted","Data":"0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab"}
Jul 27 20:19:06.751595 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.751546    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn" event={"ID":"c505919a-ad40-426e-9a96-b30eedd89231","Type":"ContainerStarted","Data":"06a6e99dab8e686472cf00f7372c476d99d4778ff0d371af20714d2a5c28ecf1"}
Jul 27 20:19:06.772502 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:06.772433    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn" podStartSLOduration=0.77241661 podStartE2EDuration="772.41661ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:19:06.770907895 +0000 UTC m=+3295.766755894" watchObservedRunningTime="2026-07-27 20:19:06.77241661 +0000 UTC m=+3295.768264587"
Jul 27 20:19:11.678575 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:11.678548    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:19:11.683517 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:11.683496    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:19:11.895499 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:11.895468    2568 scope.go:117] "RemoveContainer" containerID="436cdf971dd2e53cc5431be737f92360c0fb7e2528afc200c6ed7c7bd44409ba"
Jul 27 20:19:16.456219 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:16.456182    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:16.456219 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:16.456226    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:16.468835 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:16.468810    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:16.810628 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:16.810603    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:19:50.263851 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:50.263814    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:19:50.425296 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:50.425266    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"]
Jul 27 20:19:50.449639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:19:50.449607    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:01.475517 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.475485    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"]
Jul 27 20:20:01.475998 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.475757    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" podUID="ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" containerName="maas-api" containerID="cri-o://d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c" gracePeriod=30
Jul 27 20:20:01.498291 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.498263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:01.531946 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.531914    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"]
Jul 27 20:20:01.532164 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.532124    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc" podUID="0ac797eb-f5f0-47e8-980e-58de481eba0a" containerName="payload-processing" containerID="cri-o://e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775" gracePeriod=30
Jul 27 20:20:01.541950 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.541912    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"]
Jul 27 20:20:01.542456 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.542389    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp" podUID="eeeb4e80-d42d-4649-85ed-b08d02883175" containerName="payload-pre-processing" containerID="cri-o://397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19" gracePeriod=30
Jul 27 20:20:01.578350 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.578317    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:01.636821 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.636786    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:01.803587 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.803556    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:20:01.829543 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.829498    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:20:01.839408 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.839384    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:20:01.930915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.930882    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b8tn\" (UniqueName: \"kubernetes.io/projected/0ac797eb-f5f0-47e8-980e-58de481eba0a-kube-api-access-7b8tn\") pod \"0ac797eb-f5f0-47e8-980e-58de481eba0a\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") "
Jul 27 20:20:01.930915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.930926    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pwp4\" (UniqueName: \"kubernetes.io/projected/eeeb4e80-d42d-4649-85ed-b08d02883175-kube-api-access-9pwp4\") pod \"eeeb4e80-d42d-4649-85ed-b08d02883175\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") "
Jul 27 20:20:01.931162 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.930966    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js2xc\" (UniqueName: \"kubernetes.io/projected/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-kube-api-access-js2xc\") pod \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") "
Jul 27 20:20:01.931162 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.930996    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls\") pod \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") "
Jul 27 20:20:01.931162 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.931045    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeeb4e80-d42d-4649-85ed-b08d02883175-plugins-config-volume\") pod \"eeeb4e80-d42d-4649-85ed-b08d02883175\" (UID: \"eeeb4e80-d42d-4649-85ed-b08d02883175\") "
Jul 27 20:20:01.931162 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.931073    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac797eb-f5f0-47e8-980e-58de481eba0a-plugins-config-volume\") pod \"0ac797eb-f5f0-47e8-980e-58de481eba0a\" (UID: \"0ac797eb-f5f0-47e8-980e-58de481eba0a\") "
Jul 27 20:20:01.931162 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.931148    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-openshift-service-ca\") pod \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\" (UID: \"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b\") "
Jul 27 20:20:01.931483 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.931455    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeeb4e80-d42d-4649-85ed-b08d02883175-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eeeb4e80-d42d-4649-85ed-b08d02883175" (UID: "eeeb4e80-d42d-4649-85ed-b08d02883175"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:01.931661 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.931622    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac797eb-f5f0-47e8-980e-58de481eba0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ac797eb-f5f0-47e8-980e-58de481eba0a" (UID: "0ac797eb-f5f0-47e8-980e-58de481eba0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:01.932124 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.932097    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" (UID: "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:01.933379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.933360    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeeb4e80-d42d-4649-85ed-b08d02883175-kube-api-access-9pwp4" (OuterVolumeSpecName: "kube-api-access-9pwp4") pod "eeeb4e80-d42d-4649-85ed-b08d02883175" (UID: "eeeb4e80-d42d-4649-85ed-b08d02883175"). InnerVolumeSpecName "kube-api-access-9pwp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:01.933379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.933369    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-kube-api-access-js2xc" (OuterVolumeSpecName: "kube-api-access-js2xc") pod "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" (UID: "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b"). InnerVolumeSpecName "kube-api-access-js2xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:01.933777 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.933758    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" (UID: "ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:01.933843 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:01.933779    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac797eb-f5f0-47e8-980e-58de481eba0a-kube-api-access-7b8tn" (OuterVolumeSpecName: "kube-api-access-7b8tn") pod "0ac797eb-f5f0-47e8-980e-58de481eba0a" (UID: "0ac797eb-f5f0-47e8-980e-58de481eba0a"). InnerVolumeSpecName "kube-api-access-7b8tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:02.022917 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.022882    2568 generic.go:356] "Generic (PLEG): container finished" podID="ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" containerID="d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c" exitCode=0
Jul 27 20:20:02.023104 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.022928    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"
Jul 27 20:20:02.023104 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.022964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" event={"ID":"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b","Type":"ContainerDied","Data":"d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c"}
Jul 27 20:20:02.023104 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.023021    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h" event={"ID":"ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b","Type":"ContainerDied","Data":"c08902324e242bb07243e7634166fd0ed0a14c2879285efd4c32b0b87b4c5d44"}
Jul 27 20:20:02.023104 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.023065    2568 scope.go:117] "RemoveContainer" containerID="d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c"
Jul 27 20:20:02.024312 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.024286    2568 generic.go:356] "Generic (PLEG): container finished" podID="0ac797eb-f5f0-47e8-980e-58de481eba0a" containerID="e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775" exitCode=0
Jul 27 20:20:02.024388 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.024336    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"
Jul 27 20:20:02.024456 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.024371    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc" event={"ID":"0ac797eb-f5f0-47e8-980e-58de481eba0a","Type":"ContainerDied","Data":"e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775"}
Jul 27 20:20:02.024456 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.024426    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc" event={"ID":"0ac797eb-f5f0-47e8-980e-58de481eba0a","Type":"ContainerDied","Data":"280d1e2e119ed1074a7e11ed25dca3610657946296b9b23ac929447c90ce0d0d"}
Jul 27 20:20:02.025853 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.025830    2568 generic.go:356] "Generic (PLEG): container finished" podID="eeeb4e80-d42d-4649-85ed-b08d02883175" containerID="397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19" exitCode=0
Jul 27 20:20:02.025978 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.025860    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp" event={"ID":"eeeb4e80-d42d-4649-85ed-b08d02883175","Type":"ContainerDied","Data":"397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19"}
Jul 27 20:20:02.025978 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.025900    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp" event={"ID":"eeeb4e80-d42d-4649-85ed-b08d02883175","Type":"ContainerDied","Data":"a1e96be13564a253c446ca8e64ceee8d7a56c4ffdc009dc2b03b78755d0a62df"}
Jul 27 20:20:02.025978 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.025866    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"
Jul 27 20:20:02.032075 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032054    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwp4\" (UniqueName: \"kubernetes.io/projected/eeeb4e80-d42d-4649-85ed-b08d02883175-kube-api-access-9pwp4\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.032179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032080    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2xc\" (UniqueName: \"kubernetes.io/projected/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-kube-api-access-js2xc\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.032179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032096    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.032179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032109    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeeb4e80-d42d-4649-85ed-b08d02883175-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.032179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032123    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac797eb-f5f0-47e8-980e-58de481eba0a-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.032179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032136    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.032179 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.032150    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8tn\" (UniqueName: \"kubernetes.io/projected/0ac797eb-f5f0-47e8-980e-58de481eba0a-kube-api-access-7b8tn\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:02.033300 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.033116    2568 scope.go:117] "RemoveContainer" containerID="d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c"
Jul 27 20:20:02.033566 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:02.033405    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c\": container with ID starting with d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c not found: ID does not exist" containerID="d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c"
Jul 27 20:20:02.033566 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.033441    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c"} err="failed to get container status \"d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c\": rpc error: code = NotFound desc = could not find container \"d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c\": container with ID starting with d087b6ab62157e56da3f2ad6107ccb6d5e5d54336525f81b70923ddb67671f7c not found: ID does not exist"
Jul 27 20:20:02.033566 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.033466    2568 scope.go:117] "RemoveContainer" containerID="e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775"
Jul 27 20:20:02.049058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.049038    2568 scope.go:117] "RemoveContainer" containerID="e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775"
Jul 27 20:20:02.049360 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:02.049337    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775\": container with ID starting with e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775 not found: ID does not exist" containerID="e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775"
Jul 27 20:20:02.049442 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.049370    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775"} err="failed to get container status \"e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775\": rpc error: code = NotFound desc = could not find container \"e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775\": container with ID starting with e077fb7c177e37a94a31d68e6b72432068d6a3d7030abe89cd55092b93622775 not found: ID does not exist"
Jul 27 20:20:02.049442 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.049393    2568 scope.go:117] "RemoveContainer" containerID="397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19"
Jul 27 20:20:02.056289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.056264    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"]
Jul 27 20:20:02.057752 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.057732    2568 scope.go:117] "RemoveContainer" containerID="397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19"
Jul 27 20:20:02.058043 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:02.058026    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19\": container with ID starting with 397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19 not found: ID does not exist" containerID="397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19"
Jul 27 20:20:02.058101 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.058052    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19"} err="failed to get container status \"397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19\": rpc error: code = NotFound desc = could not find container \"397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19\": container with ID starting with 397855175bc4d375defaf21792305c6199d6dd9ca699d76c8f5185bdcbcc8d19 not found: ID does not exist"
Jul 27 20:20:02.062748 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.062726    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-71745f-6f69ccb4d9-btt6h"]
Jul 27 20:20:02.074737 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.074714    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"]
Jul 27 20:20:02.079547 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.079510    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-71745f-5f85ccb5dd-x89mp"]
Jul 27 20:20:02.092707 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.092686    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"]
Jul 27 20:20:02.097380 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:02.097358    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-71745f-79c6c97cf8-w76jc"]
Jul 27 20:20:03.568218 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:03.568147    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ac797eb-f5f0-47e8-980e-58de481eba0a" path="/var/lib/kubelet/pods/0ac797eb-f5f0-47e8-980e-58de481eba0a/volumes"
Jul 27 20:20:03.568610 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:03.568596    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" path="/var/lib/kubelet/pods/ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b/volumes"
Jul 27 20:20:03.568899 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:03.568888    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeeb4e80-d42d-4649-85ed-b08d02883175" path="/var/lib/kubelet/pods/eeeb4e80-d42d-4649-85ed-b08d02883175/volumes"
Jul 27 20:20:05.444648 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.444612    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn" podUID="c505919a-ad40-426e-9a96-b30eedd89231" containerName="main" containerID="cri-o://0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab" gracePeriod=45
Jul 27 20:20:05.628779 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.628745    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:05.698761 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.698709    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:20:05.765098 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765065    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g9f8\" (UniqueName: \"kubernetes.io/projected/c505919a-ad40-426e-9a96-b30eedd89231-kube-api-access-9g9f8\") pod \"c505919a-ad40-426e-9a96-b30eedd89231\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") "
Jul 27 20:20:05.765267 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765140    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c505919a-ad40-426e-9a96-b30eedd89231-tls-certs\") pod \"c505919a-ad40-426e-9a96-b30eedd89231\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") "
Jul 27 20:20:05.765267 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765167    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-tmp-dir\") pod \"c505919a-ad40-426e-9a96-b30eedd89231\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") "
Jul 27 20:20:05.765267 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765232    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-dshm\") pod \"c505919a-ad40-426e-9a96-b30eedd89231\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") "
Jul 27 20:20:05.765267 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765262    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-model-cache\") pod \"c505919a-ad40-426e-9a96-b30eedd89231\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") "
Jul 27 20:20:05.765483 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765312    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-home\") pod \"c505919a-ad40-426e-9a96-b30eedd89231\" (UID: \"c505919a-ad40-426e-9a96-b30eedd89231\") "
Jul 27 20:20:05.765679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765622    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c505919a-ad40-426e-9a96-b30eedd89231" (UID: "c505919a-ad40-426e-9a96-b30eedd89231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:05.765679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765660    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-model-cache" (OuterVolumeSpecName: "model-cache") pod "c505919a-ad40-426e-9a96-b30eedd89231" (UID: "c505919a-ad40-426e-9a96-b30eedd89231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:05.765679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.765671    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-home" (OuterVolumeSpecName: "home") pod "c505919a-ad40-426e-9a96-b30eedd89231" (UID: "c505919a-ad40-426e-9a96-b30eedd89231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:05.767605 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.767583    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-dshm" (OuterVolumeSpecName: "dshm") pod "c505919a-ad40-426e-9a96-b30eedd89231" (UID: "c505919a-ad40-426e-9a96-b30eedd89231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:20:05.767696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.767630    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c505919a-ad40-426e-9a96-b30eedd89231-kube-api-access-9g9f8" (OuterVolumeSpecName: "kube-api-access-9g9f8") pod "c505919a-ad40-426e-9a96-b30eedd89231" (UID: "c505919a-ad40-426e-9a96-b30eedd89231"). InnerVolumeSpecName "kube-api-access-9g9f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:05.767696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.767675    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c505919a-ad40-426e-9a96-b30eedd89231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c505919a-ad40-426e-9a96-b30eedd89231" (UID: "c505919a-ad40-426e-9a96-b30eedd89231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:05.866611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.866576    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:05.866611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.866606    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9f8\" (UniqueName: \"kubernetes.io/projected/c505919a-ad40-426e-9a96-b30eedd89231-kube-api-access-9g9f8\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:05.866611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.866616    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c505919a-ad40-426e-9a96-b30eedd89231-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:05.866844 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.866624    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:05.866844 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.866632    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:05.866844 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:05.866640    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c505919a-ad40-426e-9a96-b30eedd89231-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:06.022443 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.022411    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:06.049457 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.049430    2568 generic.go:356] "Generic (PLEG): container finished" podID="c505919a-ad40-426e-9a96-b30eedd89231" containerID="0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab" exitCode=0
Jul 27 20:20:06.049603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.049475    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"
Jul 27 20:20:06.049603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.049477    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn" event={"ID":"c505919a-ad40-426e-9a96-b30eedd89231","Type":"ContainerDied","Data":"0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab"}
Jul 27 20:20:06.049690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.049603    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn" event={"ID":"c505919a-ad40-426e-9a96-b30eedd89231","Type":"ContainerDied","Data":"06a6e99dab8e686472cf00f7372c476d99d4778ff0d371af20714d2a5c28ecf1"}
Jul 27 20:20:06.049690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.049626    2568 scope.go:117] "RemoveContainer" containerID="0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab"
Jul 27 20:20:06.059207 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.059191    2568 scope.go:117] "RemoveContainer" containerID="0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab"
Jul 27 20:20:06.059476 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:06.059456    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab\": container with ID starting with 0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab not found: ID does not exist" containerID="0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab"
Jul 27 20:20:06.059565 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.059486    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab"} err="failed to get container status \"0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab\": rpc error: code = NotFound desc = could not find container \"0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab\": container with ID starting with 0f4683e6a14625b015fe9c764e3df4a7650df89bd615530bb1f3be2c58e5e9ab not found: ID does not exist"
Jul 27 20:20:06.073011 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.072981    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"]
Jul 27 20:20:06.077649 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:06.077623    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-49ddd1/test-model-49ddd1-kserve-85b78f6f65-b7fzn"]
Jul 27 20:20:07.568457 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:07.568426    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c505919a-ad40-426e-9a96-b30eedd89231" path="/var/lib/kubelet/pods/c505919a-ad40-426e-9a96-b30eedd89231/volumes"
Jul 27 20:20:12.272511 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.272475    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:12.373349 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.373313    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:12.504859 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.504826    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:12.568056 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.567959    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:12.631811 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.631781    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:12.805591 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.805552    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:12.901886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.901800    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"]
Jul 27 20:20:12.902185 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902173    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c505919a-ad40-426e-9a96-b30eedd89231" containerName="main"
Jul 27 20:20:12.902225 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902186    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c505919a-ad40-426e-9a96-b30eedd89231" containerName="main"
Jul 27 20:20:12.902225 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902199    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac797eb-f5f0-47e8-980e-58de481eba0a" containerName="payload-processing"
Jul 27 20:20:12.902225 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902205    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac797eb-f5f0-47e8-980e-58de481eba0a" containerName="payload-processing"
Jul 27 20:20:12.902225 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902211    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeeb4e80-d42d-4649-85ed-b08d02883175" containerName="payload-pre-processing"
Jul 27 20:20:12.902225 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902218    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeeb4e80-d42d-4649-85ed-b08d02883175" containerName="payload-pre-processing"
Jul 27 20:20:12.902376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902230    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" containerName="maas-api"
Jul 27 20:20:12.902376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902235    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" containerName="maas-api"
Jul 27 20:20:12.902376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902287    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea4d8c9b-4922-4bed-b97b-5f00f26d0c0b" containerName="maas-api"
Jul 27 20:20:12.902376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902294    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeeb4e80-d42d-4649-85ed-b08d02883175" containerName="payload-pre-processing"
Jul 27 20:20:12.902376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902301    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac797eb-f5f0-47e8-980e-58de481eba0a" containerName="payload-processing"
Jul 27 20:20:12.902376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.902306    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c505919a-ad40-426e-9a96-b30eedd89231" containerName="main"
Jul 27 20:20:12.907196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.907179    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:12.909787 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.909771    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:20:12.914048 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:12.914025    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"]
Jul 27 20:20:13.034864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:13.034825    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc167c70-7c30-4ff8-9a0b-bef2317ed77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5\" (UID: \"bc167c70-7c30-4ff8-9a0b-bef2317ed77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:13.136239 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:13.136205    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc167c70-7c30-4ff8-9a0b-bef2317ed77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5\" (UID: \"bc167c70-7c30-4ff8-9a0b-bef2317ed77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:13.136858 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:13.136840    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc167c70-7c30-4ff8-9a0b-bef2317ed77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5\" (UID: \"bc167c70-7c30-4ff8-9a0b-bef2317ed77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:13.220070 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:13.219975    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:13.352634 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:13.352605    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"]
Jul 27 20:20:13.355253 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:20:13.355223    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc167c70_7c30_4ff8_9a0b_bef2317ed77e.slice/crio-fe751f96102fa6597d28909e412fb68ff3698deecd3634c88bfe74184e9c8533 WatchSource:0}: Error finding container fe751f96102fa6597d28909e412fb68ff3698deecd3634c88bfe74184e9c8533: Status 404 returned error can't find the container with id fe751f96102fa6597d28909e412fb68ff3698deecd3634c88bfe74184e9c8533
Jul 27 20:20:14.090411 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:14.090377    2568 generic.go:356] "Generic (PLEG): container finished" podID="bc167c70-7c30-4ff8-9a0b-bef2317ed77e" containerID="51a91d0de14149839300a5f5d4eed2ac92ae8cb0eea518b8fd8496db3ef1f81b" exitCode=0
Jul 27 20:20:14.090586 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:14.090420    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5" event={"ID":"bc167c70-7c30-4ff8-9a0b-bef2317ed77e","Type":"ContainerDied","Data":"51a91d0de14149839300a5f5d4eed2ac92ae8cb0eea518b8fd8496db3ef1f81b"}
Jul 27 20:20:14.090586 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:14.090463    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5" event={"ID":"bc167c70-7c30-4ff8-9a0b-bef2317ed77e","Type":"ContainerStarted","Data":"fe751f96102fa6597d28909e412fb68ff3698deecd3634c88bfe74184e9c8533"}
Jul 27 20:20:15.232805 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:15.232783    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:15.355995 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:15.355901    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc167c70-7c30-4ff8-9a0b-bef2317ed77e-maas-api-service-ca\") pod \"bc167c70-7c30-4ff8-9a0b-bef2317ed77e\" (UID: \"bc167c70-7c30-4ff8-9a0b-bef2317ed77e\") "
Jul 27 20:20:15.356298 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:15.356277    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc167c70-7c30-4ff8-9a0b-bef2317ed77e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc167c70-7c30-4ff8-9a0b-bef2317ed77e" (UID: "bc167c70-7c30-4ff8-9a0b-bef2317ed77e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:15.457616 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:15.457586    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc167c70-7c30-4ff8-9a0b-bef2317ed77e-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:16.101318 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:16.101286    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5" event={"ID":"bc167c70-7c30-4ff8-9a0b-bef2317ed77e","Type":"ContainerDied","Data":"fe751f96102fa6597d28909e412fb68ff3698deecd3634c88bfe74184e9c8533"}
Jul 27 20:20:16.101487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:16.101339    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe751f96102fa6597d28909e412fb68ff3698deecd3634c88bfe74184e9c8533"
Jul 27 20:20:16.101487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:16.101296    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"
Jul 27 20:20:20.791752 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:20.791721    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:23.500347 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.500297    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"]
Jul 27 20:20:23.500882 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.500560    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" podUID="689236a4-31db-4a29-b4ff-5137948d0897" containerName="maas-api" containerID="cri-o://c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38" gracePeriod=30
Jul 27 20:20:23.521370 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.521336    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:23.559368 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.559341    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"]
Jul 27 20:20:23.559633 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.559583    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9" podUID="4e827300-504b-4fbb-a43e-b4dddda30282" containerName="payload-processing" containerID="cri-o://2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f" gracePeriod=30
Jul 27 20:20:23.562479 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.562422    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" podUID="689236a4-31db-4a29-b4ff-5137948d0897" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.77:8443/health\": dial tcp 10.134.0.77:8443: connect: connection refused"
Jul 27 20:20:23.570983 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.570958    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"]
Jul 27 20:20:23.571189 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.571168    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" podUID="63326c4f-00ca-479e-92b3-a1313b49b0f1" containerName="payload-pre-processing" containerID="cri-o://1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69" gracePeriod=30
Jul 27 20:20:23.605867 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.605838    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:23.668834 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.668804    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:23.844744 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.844717    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:20:23.867985 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.867959    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:20:23.871553 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.871515    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:20:23.936660 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936632    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls\") pod \"689236a4-31db-4a29-b4ff-5137948d0897\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") "
Jul 27 20:20:23.936850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936669    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e827300-504b-4fbb-a43e-b4dddda30282-plugins-config-volume\") pod \"4e827300-504b-4fbb-a43e-b4dddda30282\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") "
Jul 27 20:20:23.936850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936690    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7bzz\" (UniqueName: \"kubernetes.io/projected/689236a4-31db-4a29-b4ff-5137948d0897-kube-api-access-v7bzz\") pod \"689236a4-31db-4a29-b4ff-5137948d0897\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") "
Jul 27 20:20:23.936850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936712    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/689236a4-31db-4a29-b4ff-5137948d0897-openshift-service-ca\") pod \"689236a4-31db-4a29-b4ff-5137948d0897\" (UID: \"689236a4-31db-4a29-b4ff-5137948d0897\") "
Jul 27 20:20:23.936850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936773    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63326c4f-00ca-479e-92b3-a1313b49b0f1-plugins-config-volume\") pod \"63326c4f-00ca-479e-92b3-a1313b49b0f1\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") "
Jul 27 20:20:23.936850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936789    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59dlr\" (UniqueName: \"kubernetes.io/projected/63326c4f-00ca-479e-92b3-a1313b49b0f1-kube-api-access-59dlr\") pod \"63326c4f-00ca-479e-92b3-a1313b49b0f1\" (UID: \"63326c4f-00ca-479e-92b3-a1313b49b0f1\") "
Jul 27 20:20:23.936850 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.936806    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krs4x\" (UniqueName: \"kubernetes.io/projected/4e827300-504b-4fbb-a43e-b4dddda30282-kube-api-access-krs4x\") pod \"4e827300-504b-4fbb-a43e-b4dddda30282\" (UID: \"4e827300-504b-4fbb-a43e-b4dddda30282\") "
Jul 27 20:20:23.937182 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.937110    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689236a4-31db-4a29-b4ff-5137948d0897-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "689236a4-31db-4a29-b4ff-5137948d0897" (UID: "689236a4-31db-4a29-b4ff-5137948d0897"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:23.937182 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.937164    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e827300-504b-4fbb-a43e-b4dddda30282-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e827300-504b-4fbb-a43e-b4dddda30282" (UID: "4e827300-504b-4fbb-a43e-b4dddda30282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:23.937390 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.937338    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63326c4f-00ca-479e-92b3-a1313b49b0f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63326c4f-00ca-479e-92b3-a1313b49b0f1" (UID: "63326c4f-00ca-479e-92b3-a1313b49b0f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:20:23.938911 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.938886    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689236a4-31db-4a29-b4ff-5137948d0897-kube-api-access-v7bzz" (OuterVolumeSpecName: "kube-api-access-v7bzz") pod "689236a4-31db-4a29-b4ff-5137948d0897" (UID: "689236a4-31db-4a29-b4ff-5137948d0897"). InnerVolumeSpecName "kube-api-access-v7bzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:23.938995 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.938935    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63326c4f-00ca-479e-92b3-a1313b49b0f1-kube-api-access-59dlr" (OuterVolumeSpecName: "kube-api-access-59dlr") pod "63326c4f-00ca-479e-92b3-a1313b49b0f1" (UID: "63326c4f-00ca-479e-92b3-a1313b49b0f1"). InnerVolumeSpecName "kube-api-access-59dlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:23.938995 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.938941    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "689236a4-31db-4a29-b4ff-5137948d0897" (UID: "689236a4-31db-4a29-b4ff-5137948d0897"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:20:23.939195 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:23.939177    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e827300-504b-4fbb-a43e-b4dddda30282-kube-api-access-krs4x" (OuterVolumeSpecName: "kube-api-access-krs4x") pod "4e827300-504b-4fbb-a43e-b4dddda30282" (UID: "4e827300-504b-4fbb-a43e-b4dddda30282"). InnerVolumeSpecName "kube-api-access-krs4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037833    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/689236a4-31db-4a29-b4ff-5137948d0897-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037857    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e827300-504b-4fbb-a43e-b4dddda30282-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037869    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7bzz\" (UniqueName: \"kubernetes.io/projected/689236a4-31db-4a29-b4ff-5137948d0897-kube-api-access-v7bzz\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037877    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/689236a4-31db-4a29-b4ff-5137948d0897-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037886    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63326c4f-00ca-479e-92b3-a1313b49b0f1-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037896    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59dlr\" (UniqueName: \"kubernetes.io/projected/63326c4f-00ca-479e-92b3-a1313b49b0f1-kube-api-access-59dlr\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.037904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.037904    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krs4x\" (UniqueName: \"kubernetes.io/projected/4e827300-504b-4fbb-a43e-b4dddda30282-kube-api-access-krs4x\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:20:24.143690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.143658    2568 generic.go:356] "Generic (PLEG): container finished" podID="4e827300-504b-4fbb-a43e-b4dddda30282" containerID="2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f" exitCode=0
Jul 27 20:20:24.143842 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.143704    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"
Jul 27 20:20:24.143842 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.143743    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9" event={"ID":"4e827300-504b-4fbb-a43e-b4dddda30282","Type":"ContainerDied","Data":"2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f"}
Jul 27 20:20:24.143842 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.143813    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9" event={"ID":"4e827300-504b-4fbb-a43e-b4dddda30282","Type":"ContainerDied","Data":"dfc95cb722a59b4cb9a97ecb83f38af5c9f1b841c5c0f4261d3298040286d6c8"}
Jul 27 20:20:24.144039 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.143856    2568 scope.go:117] "RemoveContainer" containerID="2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f"
Jul 27 20:20:24.145121 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.145098    2568 generic.go:356] "Generic (PLEG): container finished" podID="689236a4-31db-4a29-b4ff-5137948d0897" containerID="c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38" exitCode=0
Jul 27 20:20:24.145211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.145141    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"
Jul 27 20:20:24.145211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.145185    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" event={"ID":"689236a4-31db-4a29-b4ff-5137948d0897","Type":"ContainerDied","Data":"c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38"}
Jul 27 20:20:24.145329 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.145232    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm" event={"ID":"689236a4-31db-4a29-b4ff-5137948d0897","Type":"ContainerDied","Data":"8ab588f3c93b62b8a83dd6b3d51bf8dfd3483cd8fdb6b170dd2a0adfafc36774"}
Jul 27 20:20:24.146618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.146595    2568 generic.go:356] "Generic (PLEG): container finished" podID="63326c4f-00ca-479e-92b3-a1313b49b0f1" containerID="1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69" exitCode=0
Jul 27 20:20:24.146743 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.146672    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" event={"ID":"63326c4f-00ca-479e-92b3-a1313b49b0f1","Type":"ContainerDied","Data":"1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69"}
Jul 27 20:20:24.146743 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.146703    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"
Jul 27 20:20:24.146743 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.146710    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r" event={"ID":"63326c4f-00ca-479e-92b3-a1313b49b0f1","Type":"ContainerDied","Data":"aa1e55c89c6cdb7806869b983bea2eba5e3b452cd323a07d23adaeaa63127e9f"}
Jul 27 20:20:24.153264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.153244    2568 scope.go:117] "RemoveContainer" containerID="2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f"
Jul 27 20:20:24.153587 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:24.153516    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f\": container with ID starting with 2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f not found: ID does not exist" containerID="2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f"
Jul 27 20:20:24.153587 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.153565    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f"} err="failed to get container status \"2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f\": rpc error: code = NotFound desc = could not find container \"2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f\": container with ID starting with 2656ab8a2ed556ec45ca9ccdfec4423a60188771b15dc4c5f7aebd285aa5944f not found: ID does not exist"
Jul 27 20:20:24.153886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.153589    2568 scope.go:117] "RemoveContainer" containerID="c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38"
Jul 27 20:20:24.162645 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.162630    2568 scope.go:117] "RemoveContainer" containerID="c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38"
Jul 27 20:20:24.162919 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:24.162897    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38\": container with ID starting with c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38 not found: ID does not exist" containerID="c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38"
Jul 27 20:20:24.163007 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.162928    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38"} err="failed to get container status \"c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38\": rpc error: code = NotFound desc = could not find container \"c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38\": container with ID starting with c611a858eddfb589c6d26d5dbdba6e3ff6e57671e74d6d92ae9dee891e46fd38 not found: ID does not exist"
Jul 27 20:20:24.163007 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.162950    2568 scope.go:117] "RemoveContainer" containerID="1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69"
Jul 27 20:20:24.172925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.172908    2568 scope.go:117] "RemoveContainer" containerID="1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69"
Jul 27 20:20:24.173193 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:24.173176    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69\": container with ID starting with 1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69 not found: ID does not exist" containerID="1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69"
Jul 27 20:20:24.173258 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.173202    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69"} err="failed to get container status \"1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69\": rpc error: code = NotFound desc = could not find container \"1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69\": container with ID starting with 1e316325b4b666ac5685ea19af8ee5feca7fc6242073146c992fbf29d3feae69 not found: ID does not exist"
Jul 27 20:20:24.174542 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.174507    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"]
Jul 27 20:20:24.178282 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.178261    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-49ddd1-746c5676b8-w4l2r"]
Jul 27 20:20:24.190484 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.190460    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"]
Jul 27 20:20:24.195483 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.195463    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49ddd1-f5499f8bb-b7bpm"]
Jul 27 20:20:24.207677 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.207659    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"]
Jul 27 20:20:24.212618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:24.212600    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-49ddd1-66f9c94dc6-vdzc9"]
Jul 27 20:20:25.568080 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:25.568049    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e827300-504b-4fbb-a43e-b4dddda30282" path="/var/lib/kubelet/pods/4e827300-504b-4fbb-a43e-b4dddda30282/volumes"
Jul 27 20:20:25.568447 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:25.568372    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63326c4f-00ca-479e-92b3-a1313b49b0f1" path="/var/lib/kubelet/pods/63326c4f-00ca-479e-92b3-a1313b49b0f1/volumes"
Jul 27 20:20:25.568717 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:25.568704    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689236a4-31db-4a29-b4ff-5137948d0897" path="/var/lib/kubelet/pods/689236a4-31db-4a29-b4ff-5137948d0897/volumes"
Jul 27 20:20:27.985000 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:27.984965    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:27.991667 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:27.991642    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"]
Jul 27 20:20:27.994965 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:27.994938    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-49ddd1-z4pc5"]
Jul 27 20:20:28.363085 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:28.363043    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:29.567444 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:29.567402    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc167c70-7c30-4ff8-9a0b-bef2317ed77e" path="/var/lib/kubelet/pods/bc167c70-7c30-4ff8-9a0b-bef2317ed77e/volumes"
Jul 27 20:20:35.066111 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:35.066072    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:35.125584 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:35.125550    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:35.322298 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:35.322203    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:35.540407 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:35.540361    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:35.865415 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:35.865384    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:36.171935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:36.171840    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:36.520141 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:36.520107    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.165289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.165254    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.264447 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.264420    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.264956 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.264935    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"]
Jul 27 20:20:39.265277 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265264    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689236a4-31db-4a29-b4ff-5137948d0897" containerName="maas-api"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265278    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="689236a4-31db-4a29-b4ff-5137948d0897" containerName="maas-api"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265297    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e827300-504b-4fbb-a43e-b4dddda30282" containerName="payload-processing"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265303    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e827300-504b-4fbb-a43e-b4dddda30282" containerName="payload-processing"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265312    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc167c70-7c30-4ff8-9a0b-bef2317ed77e" containerName="revoke-keys"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265318    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc167c70-7c30-4ff8-9a0b-bef2317ed77e" containerName="revoke-keys"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265329    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63326c4f-00ca-479e-92b3-a1313b49b0f1" containerName="payload-pre-processing"
Jul 27 20:20:39.265333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265334    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="63326c4f-00ca-479e-92b3-a1313b49b0f1" containerName="payload-pre-processing"
Jul 27 20:20:39.265724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265393    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e827300-504b-4fbb-a43e-b4dddda30282" containerName="payload-processing"
Jul 27 20:20:39.265724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265401    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="63326c4f-00ca-479e-92b3-a1313b49b0f1" containerName="payload-pre-processing"
Jul 27 20:20:39.265724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265408    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="689236a4-31db-4a29-b4ff-5137948d0897" containerName="maas-api"
Jul 27 20:20:39.265724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.265416    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc167c70-7c30-4ff8-9a0b-bef2317ed77e" containerName="revoke-keys"
Jul 27 20:20:39.270089 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.270068    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.272774 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.272754    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c63843-serving-cert\""
Jul 27 20:20:39.281020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.280993    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"]
Jul 27 20:20:39.328367 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.328331    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.373512 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.373482    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec72525-ae91-475a-a1e9-00a0f20a41b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.373692 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.373544    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.373773 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.373692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f246\" (UniqueName: \"kubernetes.io/projected/0ec72525-ae91-475a-a1e9-00a0f20a41b6-kube-api-access-9f246\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.474919 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.474834    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f246\" (UniqueName: \"kubernetes.io/projected/0ec72525-ae91-475a-a1e9-00a0f20a41b6-kube-api-access-9f246\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.474919 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.474883    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec72525-ae91-475a-a1e9-00a0f20a41b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.474919 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.474913    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.475201 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:39.475025    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-serving-cert: secret "maas-api-e2e-api-a-c63843-serving-cert" not found
Jul 27 20:20:39.475201 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:20:39.475093    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls podName:0ec72525-ae91-475a-a1e9-00a0f20a41b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:20:39.975073398 +0000 UTC m=+3388.970921373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls") pod "maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" (UID: "0ec72525-ae91-475a-a1e9-00a0f20a41b6") : secret "maas-api-e2e-api-a-c63843-serving-cert" not found
Jul 27 20:20:39.475582 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.475562    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec72525-ae91-475a-a1e9-00a0f20a41b6-openshift-service-ca\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.487043 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.487020    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f246\" (UniqueName: \"kubernetes.io/projected/0ec72525-ae91-475a-a1e9-00a0f20a41b6-kube-api-access-9f246\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.529113 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.529083    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.873674 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.873640    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.971105 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.971060    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:39.980266 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.980221    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.986951 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.986913    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"]
Jul 27 20:20:39.988863 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.988839    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls\") pod \"maas-api-e2e-api-a-c63843-7796d687b7-7v4dj\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:39.991473 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.991456    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:39.994328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.994289    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c63843-dockercfg-bzhps\""
Jul 27 20:20:39.994594 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.994567    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c63843\""
Jul 27 20:20:39.998809 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:39.998789    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"]
Jul 27 20:20:40.014455 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.014432    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"]
Jul 27 20:20:40.017696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.017677    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.029898 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.029875    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"]
Jul 27 20:20:40.081496 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.081467    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wj25\" (UniqueName: \"kubernetes.io/projected/9abf1d4a-1418-445f-85de-b499bf27e592-kube-api-access-6wj25\") pod \"payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") " pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.081647 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.081518    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqkz\" (UniqueName: \"kubernetes.io/projected/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-kube-api-access-wqqkz\") pod \"payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.081647 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.081591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf1d4a-1418-445f-85de-b499bf27e592-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") " pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.081647 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.081609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.181611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.181531    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:40.182814 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.182794    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf1d4a-1418-445f-85de-b499bf27e592-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") " pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.182867 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.182824    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.182985 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.182965    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wj25\" (UniqueName: \"kubernetes.io/projected/9abf1d4a-1418-445f-85de-b499bf27e592-kube-api-access-6wj25\") pod \"payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") " pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.183054 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.183028    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqkz\" (UniqueName: \"kubernetes.io/projected/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-kube-api-access-wqqkz\") pod \"payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.183447 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.183428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.183579 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.183564    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf1d4a-1418-445f-85de-b499bf27e592-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") " pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.194205 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.194171    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wj25\" (UniqueName: \"kubernetes.io/projected/9abf1d4a-1418-445f-85de-b499bf27e592-kube-api-access-6wj25\") pod \"payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") " pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.194802 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.194778    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqkz\" (UniqueName: \"kubernetes.io/projected/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-kube-api-access-wqqkz\") pod \"payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.304167 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.304135    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:40.326980 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.326953    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:40.328409 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.328382    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"]
Jul 27 20:20:40.331498 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:20:40.331473    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec72525_ae91_475a_a1e9_00a0f20a41b6.slice/crio-153cce679b170ed9db2e8a53b91cd77239841b61382a70c34573bff9a47944ff WatchSource:0}: Error finding container 153cce679b170ed9db2e8a53b91cd77239841b61382a70c34573bff9a47944ff: Status 404 returned error can't find the container with id 153cce679b170ed9db2e8a53b91cd77239841b61382a70c34573bff9a47944ff
Jul 27 20:20:40.332852 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.332835    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:20:40.478469 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.478420    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"]
Jul 27 20:20:40.495104 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:40.495078    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"]
Jul 27 20:20:40.497518 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:20:40.497486    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abf1d4a_1418_445f_85de_b499bf27e592.slice/crio-eed9f7c2d3d5971ce778e0725bc57e4f6b0ef97a33c5ad6191d25ce234c27758 WatchSource:0}: Error finding container eed9f7c2d3d5971ce778e0725bc57e4f6b0ef97a33c5ad6191d25ce234c27758: Status 404 returned error can't find the container with id eed9f7c2d3d5971ce778e0725bc57e4f6b0ef97a33c5ad6191d25ce234c27758
Jul 27 20:20:41.232700 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.232600    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc" event={"ID":"9abf1d4a-1418-445f-85de-b499bf27e592","Type":"ContainerStarted","Data":"dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf"}
Jul 27 20:20:41.232700 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.232658    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc" event={"ID":"9abf1d4a-1418-445f-85de-b499bf27e592","Type":"ContainerStarted","Data":"eed9f7c2d3d5971ce778e0725bc57e4f6b0ef97a33c5ad6191d25ce234c27758"}
Jul 27 20:20:41.232700 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.232689    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:41.234127 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.234104    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s" event={"ID":"2f6b0d60-514d-4a48-a2bd-a03e5eb15357","Type":"ContainerStarted","Data":"a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1"}
Jul 27 20:20:41.234230 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.234152    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:41.234230 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.234176    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s" event={"ID":"2f6b0d60-514d-4a48-a2bd-a03e5eb15357","Type":"ContainerStarted","Data":"8237dfcc5305f3048e65a43ff85ebde967b4e2bc9a6afb66eef87c9a706dc1c0"}
Jul 27 20:20:41.235662 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.235639    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" event={"ID":"0ec72525-ae91-475a-a1e9-00a0f20a41b6","Type":"ContainerStarted","Data":"e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26"}
Jul 27 20:20:41.235741 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.235680    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" event={"ID":"0ec72525-ae91-475a-a1e9-00a0f20a41b6","Type":"ContainerStarted","Data":"153cce679b170ed9db2e8a53b91cd77239841b61382a70c34573bff9a47944ff"}
Jul 27 20:20:41.235741 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.235728    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:41.252166 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.252124    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc" podStartSLOduration=2.252111631 podStartE2EDuration="2.252111631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:41.250857838 +0000 UTC m=+3390.246705817" watchObservedRunningTime="2026-07-27 20:20:41.252111631 +0000 UTC m=+3390.247959608"
Jul 27 20:20:41.272987 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.272934    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s" podStartSLOduration=2.272916038 podStartE2EDuration="2.272916038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:41.270011211 +0000 UTC m=+3390.265859197" watchObservedRunningTime="2026-07-27 20:20:41.272916038 +0000 UTC m=+3390.268764019"
Jul 27 20:20:41.291827 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:41.291776    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" podStartSLOduration=1.835826588 podStartE2EDuration="2.291763847s" totalImagesPullingTime="455.937259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:41.290794511 +0000 UTC m=+3390.286642514" watchObservedRunningTime="2026-07-27 20:20:41.291763847 +0000 UTC m=+3390.287611825"
Jul 27 20:20:43.669819 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:43.669784    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:44.109482 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:44.109450    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:44.413069 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:44.412982    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:45.263698 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:45.263662    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:47.212871 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.212836    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:47.246932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.246904    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:20:47.303470 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.303441    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:47.346297 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.346264    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"]
Jul 27 20:20:47.349959 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.349943    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.352565 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.352519    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6c1463-serving-cert\""
Jul 27 20:20:47.360969 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.360949    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"]
Jul 27 20:20:47.443612 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.443581    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:47.451628 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.451597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2de16d7-37b3-4353-8041-4a30cc22dde1-maas-api-tls\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.451784 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.451635    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szd9x\" (UniqueName: \"kubernetes.io/projected/b2de16d7-37b3-4353-8041-4a30cc22dde1-kube-api-access-szd9x\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.451784 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.451694    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2de16d7-37b3-4353-8041-4a30cc22dde1-openshift-service-ca\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.552980 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.552942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2de16d7-37b3-4353-8041-4a30cc22dde1-maas-api-tls\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.552980 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.552979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szd9x\" (UniqueName: \"kubernetes.io/projected/b2de16d7-37b3-4353-8041-4a30cc22dde1-kube-api-access-szd9x\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.553204 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.553106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2de16d7-37b3-4353-8041-4a30cc22dde1-openshift-service-ca\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.553736 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.553721    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2de16d7-37b3-4353-8041-4a30cc22dde1-openshift-service-ca\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.555634 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.555614    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2de16d7-37b3-4353-8041-4a30cc22dde1-maas-api-tls\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.564132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.564107    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szd9x\" (UniqueName: \"kubernetes.io/projected/b2de16d7-37b3-4353-8041-4a30cc22dde1-kube-api-access-szd9x\") pod \"maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.579287 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.579253    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:47.660542 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.660490    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:47.797636 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:47.797598    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"]
Jul 27 20:20:47.799726 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:20:47.799702    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2de16d7_37b3_4353_8041_4a30cc22dde1.slice/crio-3c9cd59d877ee367235cf65d4007d26f7502bc44befbc2ece95e83ef751c5cc7 WatchSource:0}: Error finding container 3c9cd59d877ee367235cf65d4007d26f7502bc44befbc2ece95e83ef751c5cc7: Status 404 returned error can't find the container with id 3c9cd59d877ee367235cf65d4007d26f7502bc44befbc2ece95e83ef751c5cc7
Jul 27 20:20:48.030831 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.030794    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:48.131331 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.131246    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:48.157588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.157559    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"]
Jul 27 20:20:48.161159 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.161134    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.163785 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.163763    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6c1463\""
Jul 27 20:20:48.173580 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.173553    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"]
Jul 27 20:20:48.177131 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.177061    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"]
Jul 27 20:20:48.180818 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.180801    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.183673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.183652    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6c1463-dockercfg-smhv5\""
Jul 27 20:20:48.188848 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.188823    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"]
Jul 27 20:20:48.259568 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.259539    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a03b476-52ed-4264-8683-6430cde4d68c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.259891 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.259586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8b6t\" (UniqueName: \"kubernetes.io/projected/c4205743-7df0-4c84-adf4-9405b48ffbe7-kube-api-access-w8b6t\") pod \"payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.259891 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.259613    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nlr\" (UniqueName: \"kubernetes.io/projected/0a03b476-52ed-4264-8683-6430cde4d68c-kube-api-access-49nlr\") pod \"payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.259891 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.259712    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4205743-7df0-4c84-adf4-9405b48ffbe7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.270409 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.270382    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp" event={"ID":"b2de16d7-37b3-4353-8041-4a30cc22dde1","Type":"ContainerStarted","Data":"4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b"}
Jul 27 20:20:48.270544 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.270432    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp" event={"ID":"b2de16d7-37b3-4353-8041-4a30cc22dde1","Type":"ContainerStarted","Data":"3c9cd59d877ee367235cf65d4007d26f7502bc44befbc2ece95e83ef751c5cc7"}
Jul 27 20:20:48.270544 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.270508    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:20:48.291407 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.291352    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp" podStartSLOduration=0.922608562 podStartE2EDuration="1.291334124s" totalImagesPullingTime="368.725562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:48.28892302 +0000 UTC m=+3397.284770998" watchObservedRunningTime="2026-07-27 20:20:48.291334124 +0000 UTC m=+3397.287182103"
Jul 27 20:20:48.360450 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.360416    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a03b476-52ed-4264-8683-6430cde4d68c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.360625 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.360454    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8b6t\" (UniqueName: \"kubernetes.io/projected/c4205743-7df0-4c84-adf4-9405b48ffbe7-kube-api-access-w8b6t\") pod \"payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.360625 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.360494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nlr\" (UniqueName: \"kubernetes.io/projected/0a03b476-52ed-4264-8683-6430cde4d68c-kube-api-access-49nlr\") pod \"payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.360625 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.360612    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4205743-7df0-4c84-adf4-9405b48ffbe7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.361402 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.361371    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4205743-7df0-4c84-adf4-9405b48ffbe7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.361619 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.361600    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a03b476-52ed-4264-8683-6430cde4d68c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.370001 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.369980    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nlr\" (UniqueName: \"kubernetes.io/projected/0a03b476-52ed-4264-8683-6430cde4d68c-kube-api-access-49nlr\") pod \"payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.370141 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.370109    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8b6t\" (UniqueName: \"kubernetes.io/projected/c4205743-7df0-4c84-adf4-9405b48ffbe7-kube-api-access-w8b6t\") pod \"payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") " pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.475063 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.474983    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:48.493151 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.493121    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:48.625468 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.625285    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"]
Jul 27 20:20:48.628206 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:20:48.628180    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a03b476_52ed_4264_8683_6430cde4d68c.slice/crio-2a9dadb7928e97a34f197c8ea72770e293b8be80bdc136e846007af4d3ace336 WatchSource:0}: Error finding container 2a9dadb7928e97a34f197c8ea72770e293b8be80bdc136e846007af4d3ace336: Status 404 returned error can't find the container with id 2a9dadb7928e97a34f197c8ea72770e293b8be80bdc136e846007af4d3ace336
Jul 27 20:20:48.852022 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:48.851997    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"]
Jul 27 20:20:48.854151 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:20:48.854123    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4205743_7df0_4c84_adf4_9405b48ffbe7.slice/crio-fb171749ae029c6b2a24eaac394406dd70ef411e1333ec0d65abaa2a6db6e918 WatchSource:0}: Error finding container fb171749ae029c6b2a24eaac394406dd70ef411e1333ec0d65abaa2a6db6e918: Status 404 returned error can't find the container with id fb171749ae029c6b2a24eaac394406dd70ef411e1333ec0d65abaa2a6db6e918
Jul 27 20:20:49.277227 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.277188    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h" event={"ID":"c4205743-7df0-4c84-adf4-9405b48ffbe7","Type":"ContainerStarted","Data":"94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d"}
Jul 27 20:20:49.277654 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.277258    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:20:49.277654 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.277281    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h" event={"ID":"c4205743-7df0-4c84-adf4-9405b48ffbe7","Type":"ContainerStarted","Data":"fb171749ae029c6b2a24eaac394406dd70ef411e1333ec0d65abaa2a6db6e918"}
Jul 27 20:20:49.278816 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.278791    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j" event={"ID":"0a03b476-52ed-4264-8683-6430cde4d68c","Type":"ContainerStarted","Data":"003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab"}
Jul 27 20:20:49.278924 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.278835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j" event={"ID":"0a03b476-52ed-4264-8683-6430cde4d68c","Type":"ContainerStarted","Data":"2a9dadb7928e97a34f197c8ea72770e293b8be80bdc136e846007af4d3ace336"}
Jul 27 20:20:49.279007 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.278992    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:20:49.298664 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.298621    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h" podStartSLOduration=1.298607599 podStartE2EDuration="1.298607599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:49.29713347 +0000 UTC m=+3398.292981449" watchObservedRunningTime="2026-07-27 20:20:49.298607599 +0000 UTC m=+3398.294455577"
Jul 27 20:20:49.314768 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:49.314716    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j" podStartSLOduration=1.314698381 podStartE2EDuration="1.314698381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:20:49.312496627 +0000 UTC m=+3398.308344605" watchObservedRunningTime="2026-07-27 20:20:49.314698381 +0000 UTC m=+3398.310546362"
Jul 27 20:20:52.243690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:52.243660    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:20:52.244138 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:52.243892    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:20:52.472967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:52.472921    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:20:54.284584 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:20:54.284558    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:21:00.286264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:00.286237    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:21:00.286758 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:00.286311    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:21:11.939087 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:11.939055    2568 scope.go:117] "RemoveContainer" containerID="932f2764f8f3f1ceba6eb6f5a8d71e7896a4d03a3facc1f5a4d17f99bda9fde6"
Jul 27 20:21:14.739975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.739944    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"]
Jul 27 20:21:14.740388 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.740176    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" podUID="0ec72525-ae91-475a-a1e9-00a0f20a41b6" containerName="maas-api" containerID="cri-o://e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26" gracePeriod=30
Jul 27 20:21:14.764999 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.764974    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:14.797622 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.797590    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"]
Jul 27 20:21:14.797829 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.797805    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc" podUID="9abf1d4a-1418-445f-85de-b499bf27e592" containerName="payload-processing" containerID="cri-o://dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf" gracePeriod=30
Jul 27 20:21:14.806905 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.806877    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"]
Jul 27 20:21:14.807158 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.807115    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s" podUID="2f6b0d60-514d-4a48-a2bd-a03e5eb15357" containerName="payload-pre-processing" containerID="cri-o://a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1" gracePeriod=30
Jul 27 20:21:14.827073 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.827040    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:14.929379 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:14.929346    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:15.073657 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.073634    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:21:15.084753 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.084733    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:21:15.091299 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.091275    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:21:15.109178 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109153    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec72525-ae91-475a-a1e9-00a0f20a41b6-openshift-service-ca\") pod \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") "
Jul 27 20:21:15.109316 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109221    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f246\" (UniqueName: \"kubernetes.io/projected/0ec72525-ae91-475a-a1e9-00a0f20a41b6-kube-api-access-9f246\") pod \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") "
Jul 27 20:21:15.109316 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109269    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls\") pod \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\" (UID: \"0ec72525-ae91-475a-a1e9-00a0f20a41b6\") "
Jul 27 20:21:15.109432 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109356    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wj25\" (UniqueName: \"kubernetes.io/projected/9abf1d4a-1418-445f-85de-b499bf27e592-kube-api-access-6wj25\") pod \"9abf1d4a-1418-445f-85de-b499bf27e592\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") "
Jul 27 20:21:15.109432 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109389    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf1d4a-1418-445f-85de-b499bf27e592-plugins-config-volume\") pod \"9abf1d4a-1418-445f-85de-b499bf27e592\" (UID: \"9abf1d4a-1418-445f-85de-b499bf27e592\") "
Jul 27 20:21:15.110264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109584    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec72525-ae91-475a-a1e9-00a0f20a41b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ec72525-ae91-475a-a1e9-00a0f20a41b6" (UID: "0ec72525-ae91-475a-a1e9-00a0f20a41b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:15.110264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.109762    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec72525-ae91-475a-a1e9-00a0f20a41b6-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.110264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.110058    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abf1d4a-1418-445f-85de-b499bf27e592-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9abf1d4a-1418-445f-85de-b499bf27e592" (UID: "9abf1d4a-1418-445f-85de-b499bf27e592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:15.112354 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.112333    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ec72525-ae91-475a-a1e9-00a0f20a41b6" (UID: "0ec72525-ae91-475a-a1e9-00a0f20a41b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:21:15.112904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.112869    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abf1d4a-1418-445f-85de-b499bf27e592-kube-api-access-6wj25" (OuterVolumeSpecName: "kube-api-access-6wj25") pod "9abf1d4a-1418-445f-85de-b499bf27e592" (UID: "9abf1d4a-1418-445f-85de-b499bf27e592"). InnerVolumeSpecName "kube-api-access-6wj25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:21:15.113569 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.113543    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec72525-ae91-475a-a1e9-00a0f20a41b6-kube-api-access-9f246" (OuterVolumeSpecName: "kube-api-access-9f246") pod "0ec72525-ae91-475a-a1e9-00a0f20a41b6" (UID: "0ec72525-ae91-475a-a1e9-00a0f20a41b6"). InnerVolumeSpecName "kube-api-access-9f246". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:21:15.210876 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.210847    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqqkz\" (UniqueName: \"kubernetes.io/projected/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-kube-api-access-wqqkz\") pod \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") "
Jul 27 20:21:15.211071 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.210954    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-plugins-config-volume\") pod \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\" (UID: \"2f6b0d60-514d-4a48-a2bd-a03e5eb15357\") "
Jul 27 20:21:15.211146 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.211133    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wj25\" (UniqueName: \"kubernetes.io/projected/9abf1d4a-1418-445f-85de-b499bf27e592-kube-api-access-6wj25\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.211207 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.211148    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abf1d4a-1418-445f-85de-b499bf27e592-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.211207 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.211161    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f246\" (UniqueName: \"kubernetes.io/projected/0ec72525-ae91-475a-a1e9-00a0f20a41b6-kube-api-access-9f246\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.211207 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.211178    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec72525-ae91-475a-a1e9-00a0f20a41b6-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.211392 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.211363    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f6b0d60-514d-4a48-a2bd-a03e5eb15357" (UID: "2f6b0d60-514d-4a48-a2bd-a03e5eb15357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:15.213050 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.213028    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-kube-api-access-wqqkz" (OuterVolumeSpecName: "kube-api-access-wqqkz") pod "2f6b0d60-514d-4a48-a2bd-a03e5eb15357" (UID: "2f6b0d60-514d-4a48-a2bd-a03e5eb15357"). InnerVolumeSpecName "kube-api-access-wqqkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:21:15.312118 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.312084    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.312118 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.312114    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqkz\" (UniqueName: \"kubernetes.io/projected/2f6b0d60-514d-4a48-a2bd-a03e5eb15357-kube-api-access-wqqkz\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:15.413101 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.413072    2568 generic.go:356] "Generic (PLEG): container finished" podID="0ec72525-ae91-475a-a1e9-00a0f20a41b6" containerID="e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26" exitCode=0
Jul 27 20:21:15.413290 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.413145    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"
Jul 27 20:21:15.413290 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.413162    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" event={"ID":"0ec72525-ae91-475a-a1e9-00a0f20a41b6","Type":"ContainerDied","Data":"e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26"}
Jul 27 20:21:15.413290 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.413232    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj" event={"ID":"0ec72525-ae91-475a-a1e9-00a0f20a41b6","Type":"ContainerDied","Data":"153cce679b170ed9db2e8a53b91cd77239841b61382a70c34573bff9a47944ff"}
Jul 27 20:21:15.413290 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.413266    2568 scope.go:117] "RemoveContainer" containerID="e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26"
Jul 27 20:21:15.414880 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.414764    2568 generic.go:356] "Generic (PLEG): container finished" podID="9abf1d4a-1418-445f-85de-b499bf27e592" containerID="dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf" exitCode=0
Jul 27 20:21:15.414880 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.414806    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc" event={"ID":"9abf1d4a-1418-445f-85de-b499bf27e592","Type":"ContainerDied","Data":"dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf"}
Jul 27 20:21:15.414880 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.414818    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"
Jul 27 20:21:15.414880 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.414850    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc" event={"ID":"9abf1d4a-1418-445f-85de-b499bf27e592","Type":"ContainerDied","Data":"eed9f7c2d3d5971ce778e0725bc57e4f6b0ef97a33c5ad6191d25ce234c27758"}
Jul 27 20:21:15.416607 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.416585    2568 generic.go:356] "Generic (PLEG): container finished" podID="2f6b0d60-514d-4a48-a2bd-a03e5eb15357" containerID="a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1" exitCode=0
Jul 27 20:21:15.416607 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.416617    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"
Jul 27 20:21:15.416904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.416617    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s" event={"ID":"2f6b0d60-514d-4a48-a2bd-a03e5eb15357","Type":"ContainerDied","Data":"a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1"}
Jul 27 20:21:15.417033 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.416936    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s" event={"ID":"2f6b0d60-514d-4a48-a2bd-a03e5eb15357","Type":"ContainerDied","Data":"8237dfcc5305f3048e65a43ff85ebde967b4e2bc9a6afb66eef87c9a706dc1c0"}
Jul 27 20:21:15.423685 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.423655    2568 scope.go:117] "RemoveContainer" containerID="e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26"
Jul 27 20:21:15.424005 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:15.423974    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26\": container with ID starting with e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26 not found: ID does not exist" containerID="e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26"
Jul 27 20:21:15.424114 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.424017    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26"} err="failed to get container status \"e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26\": rpc error: code = NotFound desc = could not find container \"e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26\": container with ID starting with e0bd81a1f036c5274b7f3de4637000e09ed1fcaecf0f0cc7ef2ec756d576fc26 not found: ID does not exist"
Jul 27 20:21:15.424114 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.424038    2568 scope.go:117] "RemoveContainer" containerID="dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf"
Jul 27 20:21:15.434570 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.434553    2568 scope.go:117] "RemoveContainer" containerID="dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf"
Jul 27 20:21:15.434864 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:15.434845    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf\": container with ID starting with dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf not found: ID does not exist" containerID="dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf"
Jul 27 20:21:15.434930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.434870    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf"} err="failed to get container status \"dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf\": rpc error: code = NotFound desc = could not find container \"dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf\": container with ID starting with dc975fad5132f7ae9d86260d45f3b2984c37b46229607b8f2f8cba43a466b3cf not found: ID does not exist"
Jul 27 20:21:15.434930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.434886    2568 scope.go:117] "RemoveContainer" containerID="a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1"
Jul 27 20:21:15.443106 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.443050    2568 scope.go:117] "RemoveContainer" containerID="a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1"
Jul 27 20:21:15.443363 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:15.443344    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1\": container with ID starting with a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1 not found: ID does not exist" containerID="a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1"
Jul 27 20:21:15.443429 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.443371    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1"} err="failed to get container status \"a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1\": rpc error: code = NotFound desc = could not find container \"a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1\": container with ID starting with a3c5fc3c8e5ff27079514a2a3fe5ca06ab21f33805fcfa07985ebcb0d5b7d4b1 not found: ID does not exist"
Jul 27 20:21:15.445580 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.445560    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"]
Jul 27 20:21:15.449726 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.449705    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c63843-7796d687b7-7v4dj"]
Jul 27 20:21:15.461280 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.461250    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"]
Jul 27 20:21:15.467313 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.467295    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c63843-65bcc6664b-qs6vc"]
Jul 27 20:21:15.479268 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.479248    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"]
Jul 27 20:21:15.480968 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.480949    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c63843-6fd97f47b4-9p55s"]
Jul 27 20:21:15.567891 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.567813    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec72525-ae91-475a-a1e9-00a0f20a41b6" path="/var/lib/kubelet/pods/0ec72525-ae91-475a-a1e9-00a0f20a41b6/volumes"
Jul 27 20:21:15.568186 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.568173    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f6b0d60-514d-4a48-a2bd-a03e5eb15357" path="/var/lib/kubelet/pods/2f6b0d60-514d-4a48-a2bd-a03e5eb15357/volumes"
Jul 27 20:21:15.568483 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:15.568472    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abf1d4a-1418-445f-85de-b499bf27e592" path="/var/lib/kubelet/pods/9abf1d4a-1418-445f-85de-b499bf27e592/volumes"
Jul 27 20:21:19.794610 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:19.794576    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:20.208127 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:20.208032    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:26.755015 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:26.754975    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:26.821935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:26.821903    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:27.017078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.016988    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:27.237998 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.237965    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:27.364373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364343    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"]
Jul 27 20:21:27.364734 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364721    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f6b0d60-514d-4a48-a2bd-a03e5eb15357" containerName="payload-pre-processing"
Jul 27 20:21:27.364789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364736    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f6b0d60-514d-4a48-a2bd-a03e5eb15357" containerName="payload-pre-processing"
Jul 27 20:21:27.364789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364761    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abf1d4a-1418-445f-85de-b499bf27e592" containerName="payload-processing"
Jul 27 20:21:27.364789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364767    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abf1d4a-1418-445f-85de-b499bf27e592" containerName="payload-processing"
Jul 27 20:21:27.364789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364778    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec72525-ae91-475a-a1e9-00a0f20a41b6" containerName="maas-api"
Jul 27 20:21:27.364789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364784    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec72525-ae91-475a-a1e9-00a0f20a41b6" containerName="maas-api"
Jul 27 20:21:27.364937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364840    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abf1d4a-1418-445f-85de-b499bf27e592" containerName="payload-processing"
Jul 27 20:21:27.364937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364847    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f6b0d60-514d-4a48-a2bd-a03e5eb15357" containerName="payload-pre-processing"
Jul 27 20:21:27.364937 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.364853    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec72525-ae91-475a-a1e9-00a0f20a41b6" containerName="maas-api"
Jul 27 20:21:27.368060 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.368041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:27.370725 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.370707    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:21:27.377701 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.377679    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"]
Jul 27 20:21:27.423678 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.423641    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg\" (UID: \"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:27.524932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.524899    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg\" (UID: \"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:27.525629 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.525609    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg\" (UID: \"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:27.679855 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.679773    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:27.811432 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:27.811409    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"]
Jul 27 20:21:27.814193 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:21:27.814167    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdcdf18_61d4_4138_b3db_68a3e2b2b8de.slice/crio-87b060dc8255bf3c310efce453733f36fa0915ccf0630dd21a3f4b49181b4bb5 WatchSource:0}: Error finding container 87b060dc8255bf3c310efce453733f36fa0915ccf0630dd21a3f4b49181b4bb5: Status 404 returned error can't find the container with id 87b060dc8255bf3c310efce453733f36fa0915ccf0630dd21a3f4b49181b4bb5
Jul 27 20:21:28.489924 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:28.489893    2568 generic.go:356] "Generic (PLEG): container finished" podID="5bdcdf18-61d4-4138-b3db-68a3e2b2b8de" containerID="dc95a7a0b8825bb096af997c43e5fb3a7cdd35662a502ad76a0297f6190712dd" exitCode=0
Jul 27 20:21:28.490078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:28.489933    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg" event={"ID":"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de","Type":"ContainerDied","Data":"dc95a7a0b8825bb096af997c43e5fb3a7cdd35662a502ad76a0297f6190712dd"}
Jul 27 20:21:28.490078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:28.489973    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg" event={"ID":"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de","Type":"ContainerStarted","Data":"87b060dc8255bf3c310efce453733f36fa0915ccf0630dd21a3f4b49181b4bb5"}
Jul 27 20:21:29.626753 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:29.626727    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:29.745738 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:29.745654    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de-maas-api-service-ca\") pod \"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de\" (UID: \"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de\") "
Jul 27 20:21:29.746006 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:29.745983    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bdcdf18-61d4-4138-b3db-68a3e2b2b8de" (UID: "5bdcdf18-61d4-4138-b3db-68a3e2b2b8de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:29.846960 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:29.846932    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:30.500804 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:30.500777    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"
Jul 27 20:21:30.500985 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:30.500826    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg" event={"ID":"5bdcdf18-61d4-4138-b3db-68a3e2b2b8de","Type":"ContainerDied","Data":"87b060dc8255bf3c310efce453733f36fa0915ccf0630dd21a3f4b49181b4bb5"}
Jul 27 20:21:30.500985 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:30.500874    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b060dc8255bf3c310efce453733f36fa0915ccf0630dd21a3f4b49181b4bb5"
Jul 27 20:21:42.806980 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.806944    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"]
Jul 27 20:21:42.807571 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.807216    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp" podUID="b2de16d7-37b3-4353-8041-4a30cc22dde1" containerName="maas-api" containerID="cri-o://4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b" gracePeriod=30
Jul 27 20:21:42.830984 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.830954    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:42.867575 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.867542    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"]
Jul 27 20:21:42.867820 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.867798    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h" podUID="c4205743-7df0-4c84-adf4-9405b48ffbe7" containerName="payload-processing" containerID="cri-o://94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d" gracePeriod=30
Jul 27 20:21:42.876992 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.876753    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"]
Jul 27 20:21:42.877163 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.876989    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j" podUID="0a03b476-52ed-4264-8683-6430cde4d68c" containerName="payload-pre-processing" containerID="cri-o://003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab" gracePeriod=30
Jul 27 20:21:42.917020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.916979    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:42.974122 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:42.974084    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:43.157954 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.157929    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:21:43.161151 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.161134    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:21:43.166518 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.166500    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:21:43.179685 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179664    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2de16d7-37b3-4353-8041-4a30cc22dde1-maas-api-tls\") pod \"b2de16d7-37b3-4353-8041-4a30cc22dde1\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") "
Jul 27 20:21:43.179815 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179694    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2de16d7-37b3-4353-8041-4a30cc22dde1-openshift-service-ca\") pod \"b2de16d7-37b3-4353-8041-4a30cc22dde1\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") "
Jul 27 20:21:43.179815 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179718    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a03b476-52ed-4264-8683-6430cde4d68c-plugins-config-volume\") pod \"0a03b476-52ed-4264-8683-6430cde4d68c\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") "
Jul 27 20:21:43.179815 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179742    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49nlr\" (UniqueName: \"kubernetes.io/projected/0a03b476-52ed-4264-8683-6430cde4d68c-kube-api-access-49nlr\") pod \"0a03b476-52ed-4264-8683-6430cde4d68c\" (UID: \"0a03b476-52ed-4264-8683-6430cde4d68c\") "
Jul 27 20:21:43.179995 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179815    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szd9x\" (UniqueName: \"kubernetes.io/projected/b2de16d7-37b3-4353-8041-4a30cc22dde1-kube-api-access-szd9x\") pod \"b2de16d7-37b3-4353-8041-4a30cc22dde1\" (UID: \"b2de16d7-37b3-4353-8041-4a30cc22dde1\") "
Jul 27 20:21:43.179995 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179832    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4205743-7df0-4c84-adf4-9405b48ffbe7-plugins-config-volume\") pod \"c4205743-7df0-4c84-adf4-9405b48ffbe7\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") "
Jul 27 20:21:43.179995 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.179855    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8b6t\" (UniqueName: \"kubernetes.io/projected/c4205743-7df0-4c84-adf4-9405b48ffbe7-kube-api-access-w8b6t\") pod \"c4205743-7df0-4c84-adf4-9405b48ffbe7\" (UID: \"c4205743-7df0-4c84-adf4-9405b48ffbe7\") "
Jul 27 20:21:43.180192 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.180087    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2de16d7-37b3-4353-8041-4a30cc22dde1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2de16d7-37b3-4353-8041-4a30cc22dde1" (UID: "b2de16d7-37b3-4353-8041-4a30cc22dde1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:43.180382 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.180327    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4205743-7df0-4c84-adf4-9405b48ffbe7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4205743-7df0-4c84-adf4-9405b48ffbe7" (UID: "c4205743-7df0-4c84-adf4-9405b48ffbe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:43.180505 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.180387    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a03b476-52ed-4264-8683-6430cde4d68c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a03b476-52ed-4264-8683-6430cde4d68c" (UID: "0a03b476-52ed-4264-8683-6430cde4d68c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:21:43.182893 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.182856    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a03b476-52ed-4264-8683-6430cde4d68c-kube-api-access-49nlr" (OuterVolumeSpecName: "kube-api-access-49nlr") pod "0a03b476-52ed-4264-8683-6430cde4d68c" (UID: "0a03b476-52ed-4264-8683-6430cde4d68c"). InnerVolumeSpecName "kube-api-access-49nlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:21:43.182893 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.182877    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2de16d7-37b3-4353-8041-4a30cc22dde1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2de16d7-37b3-4353-8041-4a30cc22dde1" (UID: "b2de16d7-37b3-4353-8041-4a30cc22dde1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:21:43.183077 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.182893    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2de16d7-37b3-4353-8041-4a30cc22dde1-kube-api-access-szd9x" (OuterVolumeSpecName: "kube-api-access-szd9x") pod "b2de16d7-37b3-4353-8041-4a30cc22dde1" (UID: "b2de16d7-37b3-4353-8041-4a30cc22dde1"). InnerVolumeSpecName "kube-api-access-szd9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:21:43.183077 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.182909    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4205743-7df0-4c84-adf4-9405b48ffbe7-kube-api-access-w8b6t" (OuterVolumeSpecName: "kube-api-access-w8b6t") pod "c4205743-7df0-4c84-adf4-9405b48ffbe7" (UID: "c4205743-7df0-4c84-adf4-9405b48ffbe7"). InnerVolumeSpecName "kube-api-access-w8b6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:21:43.281097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281065    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szd9x\" (UniqueName: \"kubernetes.io/projected/b2de16d7-37b3-4353-8041-4a30cc22dde1-kube-api-access-szd9x\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.281097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281089    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4205743-7df0-4c84-adf4-9405b48ffbe7-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.281097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281098    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8b6t\" (UniqueName: \"kubernetes.io/projected/c4205743-7df0-4c84-adf4-9405b48ffbe7-kube-api-access-w8b6t\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.281097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281108    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2de16d7-37b3-4353-8041-4a30cc22dde1-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.281363 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281117    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2de16d7-37b3-4353-8041-4a30cc22dde1-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.281363 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281127    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a03b476-52ed-4264-8683-6430cde4d68c-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.281363 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.281135    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nlr\" (UniqueName: \"kubernetes.io/projected/0a03b476-52ed-4264-8683-6430cde4d68c-kube-api-access-49nlr\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:21:43.563993 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.563951    2568 generic.go:356] "Generic (PLEG): container finished" podID="c4205743-7df0-4c84-adf4-9405b48ffbe7" containerID="94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d" exitCode=0
Jul 27 20:21:43.564200 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.564055    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"
Jul 27 20:21:43.565383 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.565358    2568 generic.go:356] "Generic (PLEG): container finished" podID="0a03b476-52ed-4264-8683-6430cde4d68c" containerID="003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab" exitCode=0
Jul 27 20:21:43.565546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.565458    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"
Jul 27 20:21:43.566833 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.566810    2568 generic.go:356] "Generic (PLEG): container finished" podID="b2de16d7-37b3-4353-8041-4a30cc22dde1" containerID="4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b" exitCode=0
Jul 27 20:21:43.566968 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.566924    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"
Jul 27 20:21:43.568116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h" event={"ID":"c4205743-7df0-4c84-adf4-9405b48ffbe7","Type":"ContainerDied","Data":"94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d"}
Jul 27 20:21:43.568247 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h" event={"ID":"c4205743-7df0-4c84-adf4-9405b48ffbe7","Type":"ContainerDied","Data":"fb171749ae029c6b2a24eaac394406dd70ef411e1333ec0d65abaa2a6db6e918"}
Jul 27 20:21:43.568247 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568166    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j" event={"ID":"0a03b476-52ed-4264-8683-6430cde4d68c","Type":"ContainerDied","Data":"003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab"}
Jul 27 20:21:43.568247 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568190    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j" event={"ID":"0a03b476-52ed-4264-8683-6430cde4d68c","Type":"ContainerDied","Data":"2a9dadb7928e97a34f197c8ea72770e293b8be80bdc136e846007af4d3ace336"}
Jul 27 20:21:43.568247 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568210    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp" event={"ID":"b2de16d7-37b3-4353-8041-4a30cc22dde1","Type":"ContainerDied","Data":"4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b"}
Jul 27 20:21:43.568247 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568236    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp" event={"ID":"b2de16d7-37b3-4353-8041-4a30cc22dde1","Type":"ContainerDied","Data":"3c9cd59d877ee367235cf65d4007d26f7502bc44befbc2ece95e83ef751c5cc7"}
Jul 27 20:21:43.568546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.568264    2568 scope.go:117] "RemoveContainer" containerID="94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d"
Jul 27 20:21:43.580372 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.580337    2568 scope.go:117] "RemoveContainer" containerID="94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d"
Jul 27 20:21:43.580811 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:43.580784    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d\": container with ID starting with 94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d not found: ID does not exist" containerID="94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d"
Jul 27 20:21:43.580910 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.580823    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d"} err="failed to get container status \"94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d\": rpc error: code = NotFound desc = could not find container \"94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d\": container with ID starting with 94fa0a71a5c41a22ceda36f22e5d5077449e81b4ab0c52b55ab824e34b69368d not found: ID does not exist"
Jul 27 20:21:43.580910 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.580849    2568 scope.go:117] "RemoveContainer" containerID="003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab"
Jul 27 20:21:43.589318 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.589299    2568 scope.go:117] "RemoveContainer" containerID="003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab"
Jul 27 20:21:43.589577 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:43.589557    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab\": container with ID starting with 003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab not found: ID does not exist" containerID="003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab"
Jul 27 20:21:43.589634 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.589586    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab"} err="failed to get container status \"003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab\": rpc error: code = NotFound desc = could not find container \"003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab\": container with ID starting with 003618aa063b62323e9a1ecd89888e95b1d55afe5e725e6f2e1c0446be67cfab not found: ID does not exist"
Jul 27 20:21:43.589634 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.589602    2568 scope.go:117] "RemoveContainer" containerID="4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b"
Jul 27 20:21:43.598394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.598360    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"]
Jul 27 20:21:43.600032 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.600011    2568 scope.go:117] "RemoveContainer" containerID="4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b"
Jul 27 20:21:43.600671 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:43.600640    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b\": container with ID starting with 4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b not found: ID does not exist" containerID="4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b"
Jul 27 20:21:43.600772 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.600681    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b"} err="failed to get container status \"4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b\": rpc error: code = NotFound desc = could not find container \"4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b\": container with ID starting with 4c87ec70d1ce87b51e10939a5288450cb988d4a97246cec30e062b638c87422b not found: ID does not exist"
Jul 27 20:21:43.603038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.603007    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6c1463-66c99ff65b-lmh8j"]
Jul 27 20:21:43.612696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.612676    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"]
Jul 27 20:21:43.617272 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.617251    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6c1463-66b5b49858-wfbcp"]
Jul 27 20:21:43.629406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.629388    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"]
Jul 27 20:21:43.635094 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:43.635072    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6c1463-5db867cb96-ftq2h"]
Jul 27 20:21:45.569263 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:45.569231    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a03b476-52ed-4264-8683-6430cde4d68c" path="/var/lib/kubelet/pods/0a03b476-52ed-4264-8683-6430cde4d68c/volumes"
Jul 27 20:21:45.569651 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:45.569600    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2de16d7-37b3-4353-8041-4a30cc22dde1" path="/var/lib/kubelet/pods/b2de16d7-37b3-4353-8041-4a30cc22dde1/volumes"
Jul 27 20:21:45.569915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:45.569902    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4205743-7df0-4c84-adf4-9405b48ffbe7" path="/var/lib/kubelet/pods/c4205743-7df0-4c84-adf4-9405b48ffbe7/volumes"
Jul 27 20:21:47.442819 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:47.442787    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:47.449960 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:47.449932    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"]
Jul 27 20:21:47.452662 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:47.452639    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6c1463-h6qsg"]
Jul 27 20:21:47.568603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:47.568481    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bdcdf18-61d4-4138-b3db-68a3e2b2b8de" path="/var/lib/kubelet/pods/5bdcdf18-61d4-4138-b3db-68a3e2b2b8de/volumes"
Jul 27 20:21:47.870907 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:47.870872    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:54.210773 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:54.210736    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:54.380306 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:54.380263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:54.482328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:54.482219    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:54.696769 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:54.696741    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:55.024967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:55.024930    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:55.352030 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:55.351932    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:55.533317 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:55.533280    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:58.572967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.572929    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:58.656653 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.656614    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"]
Jul 27 20:21:58.656975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.656961    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4205743-7df0-4c84-adf4-9405b48ffbe7" containerName="payload-processing"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.656976    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4205743-7df0-4c84-adf4-9405b48ffbe7" containerName="payload-processing"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.656998    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a03b476-52ed-4264-8683-6430cde4d68c" containerName="payload-pre-processing"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657004    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a03b476-52ed-4264-8683-6430cde4d68c" containerName="payload-pre-processing"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657016    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2de16d7-37b3-4353-8041-4a30cc22dde1" containerName="maas-api"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657025    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2de16d7-37b3-4353-8041-4a30cc22dde1" containerName="maas-api"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657048    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdcdf18-61d4-4138-b3db-68a3e2b2b8de" containerName="revoke-keys"
Jul 27 20:21:58.657058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657056    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdcdf18-61d4-4138-b3db-68a3e2b2b8de" containerName="revoke-keys"
Jul 27 20:21:58.657427 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657111    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4205743-7df0-4c84-adf4-9405b48ffbe7" containerName="payload-processing"
Jul 27 20:21:58.657427 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657127    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdcdf18-61d4-4138-b3db-68a3e2b2b8de" containerName="revoke-keys"
Jul 27 20:21:58.657427 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657138    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a03b476-52ed-4264-8683-6430cde4d68c" containerName="payload-pre-processing"
Jul 27 20:21:58.657427 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657147    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2de16d7-37b3-4353-8041-4a30cc22dde1" containerName="maas-api"
Jul 27 20:21:58.657767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.657738    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:58.660310 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.660296    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.662855 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.662833    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a78f44-serving-cert\""
Jul 27 20:21:58.671650 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.671629    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"]
Jul 27 20:21:58.724416 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.724371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlxm\" (UniqueName: \"kubernetes.io/projected/5a0da973-1c0c-4e94-b833-2b06e68a117b-kube-api-access-drlxm\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.724416 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.724416    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0da973-1c0c-4e94-b833-2b06e68a117b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.724766 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.724471    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.724766 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.724582    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:58.825185 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.825089    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlxm\" (UniqueName: \"kubernetes.io/projected/5a0da973-1c0c-4e94-b833-2b06e68a117b-kube-api-access-drlxm\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.825185 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.825134    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0da973-1c0c-4e94-b833-2b06e68a117b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.825185 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.825174    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.825408 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:58.825278    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-serving-cert: secret "maas-api-e2e-shared-a-a78f44-serving-cert" not found
Jul 27 20:21:58.825408 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:21:58.825343    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls podName:5a0da973-1c0c-4e94-b833-2b06e68a117b nodeName:}" failed. No retries permitted until 2026-07-27 20:21:59.325325487 +0000 UTC m=+3468.321173446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls") pod "maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" (UID: "5a0da973-1c0c-4e94-b833-2b06e68a117b") : secret "maas-api-e2e-shared-a-a78f44-serving-cert" not found
Jul 27 20:21:58.825827 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.825807    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0da973-1c0c-4e94-b833-2b06e68a117b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.835488 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.835460    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlxm\" (UniqueName: \"kubernetes.io/projected/5a0da973-1c0c-4e94-b833-2b06e68a117b-kube-api-access-drlxm\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:58.924050 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:58.924013    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:59.248922 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.248829    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:59.329929 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.329892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:59.332411 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.332381    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:59.351068 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.351029    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:21:59.351345 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.351327    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"]
Jul 27 20:21:59.354810 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.354796    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.357612 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.357594    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a78f44\""
Jul 27 20:21:59.357690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.357658    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a78f44-dockercfg-jfmlg\""
Jul 27 20:21:59.365351 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.365331    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"]
Jul 27 20:21:59.387703 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.387676    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"]
Jul 27 20:21:59.390990 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.390975    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.401975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.401952    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"]
Jul 27 20:21:59.431020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.430985    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-kube-api-access-8zrn4\") pod \"payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.431171 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.431049    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.531877 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.531842    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-kube-api-access-8zrn4\") pod \"payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.532029 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.531885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09afcc06-b883-4b85-b631-dd0eb9007d76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.532029 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.531922    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn4q\" (UniqueName: \"kubernetes.io/projected/09afcc06-b883-4b85-b631-dd0eb9007d76-kube-api-access-rdn4q\") pod \"payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.532029 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.531952    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.532668 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.532646    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.543311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.543287    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-kube-api-access-8zrn4\") pod \"payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.570573 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.570542    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:21:59.632889 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.632820    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09afcc06-b883-4b85-b631-dd0eb9007d76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.634406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.633782    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09afcc06-b883-4b85-b631-dd0eb9007d76-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.634406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.634293    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn4q\" (UniqueName: \"kubernetes.io/projected/09afcc06-b883-4b85-b631-dd0eb9007d76-kube-api-access-rdn4q\") pod \"payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.643915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.643884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn4q\" (UniqueName: \"kubernetes.io/projected/09afcc06-b883-4b85-b631-dd0eb9007d76-kube-api-access-rdn4q\") pod \"payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.665498 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.665474    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:21:59.700704 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.700641    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:21:59.714377 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.714352    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"]
Jul 27 20:21:59.716682 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:21:59.716650    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0da973_1c0c_4e94_b833_2b06e68a117b.slice/crio-7fcb4ba7f21bb0d8b1571b027e6f6caa8f4c68b05997644816068554f69e5708 WatchSource:0}: Error finding container 7fcb4ba7f21bb0d8b1571b027e6f6caa8f4c68b05997644816068554f69e5708: Status 404 returned error can't find the container with id 7fcb4ba7f21bb0d8b1571b027e6f6caa8f4c68b05997644816068554f69e5708
Jul 27 20:21:59.839405 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.839375    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"]
Jul 27 20:21:59.841057 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:21:59.841028    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181a332a_b79e_4d9b_b9ca_daf8d2c9cbd7.slice/crio-5655fe56d5002b0395f85185c3a911080b18b7be6ff1eca0874675035f4b7dd0 WatchSource:0}: Error finding container 5655fe56d5002b0395f85185c3a911080b18b7be6ff1eca0874675035f4b7dd0: Status 404 returned error can't find the container with id 5655fe56d5002b0395f85185c3a911080b18b7be6ff1eca0874675035f4b7dd0
Jul 27 20:21:59.865823 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:21:59.865799    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"]
Jul 27 20:21:59.868684 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:21:59.868655    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09afcc06_b883_4b85_b631_dd0eb9007d76.slice/crio-001c21b9097705558c2ece92138cde14260cc0f4c824afc0b61ec6e199671cda WatchSource:0}: Error finding container 001c21b9097705558c2ece92138cde14260cc0f4c824afc0b61ec6e199671cda: Status 404 returned error can't find the container with id 001c21b9097705558c2ece92138cde14260cc0f4c824afc0b61ec6e199671cda
Jul 27 20:22:00.662613 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.662565    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk" event={"ID":"09afcc06-b883-4b85-b631-dd0eb9007d76","Type":"ContainerStarted","Data":"83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473"}
Jul 27 20:22:00.663250 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.662630    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk" event={"ID":"09afcc06-b883-4b85-b631-dd0eb9007d76","Type":"ContainerStarted","Data":"001c21b9097705558c2ece92138cde14260cc0f4c824afc0b61ec6e199671cda"}
Jul 27 20:22:00.663250 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.662688    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:22:00.664251 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.664220    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" event={"ID":"5a0da973-1c0c-4e94-b833-2b06e68a117b","Type":"ContainerStarted","Data":"c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635"}
Jul 27 20:22:00.664387 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.664266    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:22:00.664387 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.664287    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" event={"ID":"5a0da973-1c0c-4e94-b833-2b06e68a117b","Type":"ContainerStarted","Data":"7fcb4ba7f21bb0d8b1571b027e6f6caa8f4c68b05997644816068554f69e5708"}
Jul 27 20:22:00.665819 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.665794    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2" event={"ID":"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7","Type":"ContainerStarted","Data":"93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2"}
Jul 27 20:22:00.665924 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.665835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2" event={"ID":"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7","Type":"ContainerStarted","Data":"5655fe56d5002b0395f85185c3a911080b18b7be6ff1eca0874675035f4b7dd0"}
Jul 27 20:22:00.665924 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.665894    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:22:00.683362 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.683313    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk" podStartSLOduration=1.6832986829999999 podStartE2EDuration="1.683298683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:00.682076826 +0000 UTC m=+3469.677924805" watchObservedRunningTime="2026-07-27 20:22:00.683298683 +0000 UTC m=+3469.679146652"
Jul 27 20:22:00.702057 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.702001    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" podStartSLOduration=2.196600405 podStartE2EDuration="2.70198265s" totalImagesPullingTime="505.382245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:00.701641937 +0000 UTC m=+3469.697489934" watchObservedRunningTime="2026-07-27 20:22:00.70198265 +0000 UTC m=+3469.697830630"
Jul 27 20:22:00.722944 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:00.722897    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2" podStartSLOduration=1.72288457 podStartE2EDuration="1.72288457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:00.722102501 +0000 UTC m=+3469.717950520" watchObservedRunningTime="2026-07-27 20:22:00.72288457 +0000 UTC m=+3469.718732575"
Jul 27 20:22:02.995728 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:02.995691    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:03.115839 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:03.115798    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:03.372660 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:03.372598    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:03.664316 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:03.664238    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:04.076900 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:04.076867    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:06.676564 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.676507    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:22:06.729811 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.729781    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"]
Jul 27 20:22:06.733662 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.733638    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.736432 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.736411    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-43d589-serving-cert\""
Jul 27 20:22:06.745097 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.745073    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"]
Jul 27 20:22:06.799638 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.799598    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.799809 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.799662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bps\" (UniqueName: \"kubernetes.io/projected/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-kube-api-access-47bps\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.799809 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.799791    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.901020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.900986    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.901202 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.901044    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bps\" (UniqueName: \"kubernetes.io/projected/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-kube-api-access-47bps\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.901202 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.901099    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.901202 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:22:06.901192    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-serving-cert: secret "maas-api-e2e-shared-b-43d589-serving-cert" not found
Jul 27 20:22:06.901307 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:22:06.901254    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls podName:9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b nodeName:}" failed. No retries permitted until 2026-07-27 20:22:07.401238847 +0000 UTC m=+3476.397086803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls") pod "maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" (UID: "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b") : secret "maas-api-e2e-shared-b-43d589-serving-cert" not found
Jul 27 20:22:06.901669 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.901650    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.915230 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.915205    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bps\" (UniqueName: \"kubernetes.io/projected/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-kube-api-access-47bps\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:06.950428 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:06.950345    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:07.063442 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.063402    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:07.201411 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.201321    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:07.406334 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.406300    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:07.409264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.409236    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:07.445788 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.445750    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:07.539906 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.539875    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:07.541194 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.541164    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"]
Jul 27 20:22:07.544950 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.544931    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.548194 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.548174    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-43d589\""
Jul 27 20:22:07.555061 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.555037    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"]
Jul 27 20:22:07.587496 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.587461    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"]
Jul 27 20:22:07.590880 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.590861    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.593633 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.593616    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-43d589-dockercfg-mtl9q\""
Jul 27 20:22:07.601387 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.601362    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"]
Jul 27 20:22:07.646480 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.646448    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:07.709132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.709099    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsc4\" (UniqueName: \"kubernetes.io/projected/d339a66c-c034-46c1-ae27-cc4110066d0d-kube-api-access-tcsc4\") pod \"payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.709562 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.709170    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcqz\" (UniqueName: \"kubernetes.io/projected/a339ace1-88ea-441f-8306-daad94f35ec4-kube-api-access-bwcqz\") pod \"payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.709562 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.709223    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339ace1-88ea-441f-8306-daad94f35ec4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.709562 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.709284    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d339a66c-c034-46c1-ae27-cc4110066d0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.788188 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.788159    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"]
Jul 27 20:22:07.790317 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:22:07.790234    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcd6d6e_9c96_4d26_ad0c_3575d3cb6a8b.slice/crio-912ce7c902744a9da5582d869179071f6013fd3c44d17fbd93e65148fa0eea51 WatchSource:0}: Error finding container 912ce7c902744a9da5582d869179071f6013fd3c44d17fbd93e65148fa0eea51: Status 404 returned error can't find the container with id 912ce7c902744a9da5582d869179071f6013fd3c44d17fbd93e65148fa0eea51
Jul 27 20:22:07.810778 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.810751    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d339a66c-c034-46c1-ae27-cc4110066d0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.810883 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.810837    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsc4\" (UniqueName: \"kubernetes.io/projected/d339a66c-c034-46c1-ae27-cc4110066d0d-kube-api-access-tcsc4\") pod \"payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.810925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.810884    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcqz\" (UniqueName: \"kubernetes.io/projected/a339ace1-88ea-441f-8306-daad94f35ec4-kube-api-access-bwcqz\") pod \"payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.810961 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.810932    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339ace1-88ea-441f-8306-daad94f35ec4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.811476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.811459    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d339a66c-c034-46c1-ae27-cc4110066d0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.811622 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.811606    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339ace1-88ea-441f-8306-daad94f35ec4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.821657 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.821630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsc4\" (UniqueName: \"kubernetes.io/projected/d339a66c-c034-46c1-ae27-cc4110066d0d-kube-api-access-tcsc4\") pod \"payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:07.821746 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.821627    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcqz\" (UniqueName: \"kubernetes.io/projected/a339ace1-88ea-441f-8306-daad94f35ec4-kube-api-access-bwcqz\") pod \"payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.856504 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.856479    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:07.901564 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:07.901502    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:08.009791 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.009749    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"]
Jul 27 20:22:08.012203 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:22:08.012173    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda339ace1_88ea_441f_8306_daad94f35ec4.slice/crio-22b2ce2aa35c6f52e2f5d53ff47b3fafb85d9e0d59821a843c2b894eb82a4bae WatchSource:0}: Error finding container 22b2ce2aa35c6f52e2f5d53ff47b3fafb85d9e0d59821a843c2b894eb82a4bae: Status 404 returned error can't find the container with id 22b2ce2aa35c6f52e2f5d53ff47b3fafb85d9e0d59821a843c2b894eb82a4bae
Jul 27 20:22:08.064015 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.063993    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"]
Jul 27 20:22:08.066201 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:22:08.066176    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd339a66c_c034_46c1_ae27_cc4110066d0d.slice/crio-c3b49f986358c0388225d726d33f3459bdecf72d87dd144d46be76db0ecf9705 WatchSource:0}: Error finding container c3b49f986358c0388225d726d33f3459bdecf72d87dd144d46be76db0ecf9705: Status 404 returned error can't find the container with id c3b49f986358c0388225d726d33f3459bdecf72d87dd144d46be76db0ecf9705
Jul 27 20:22:08.716132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.716047    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6" event={"ID":"d339a66c-c034-46c1-ae27-cc4110066d0d","Type":"ContainerStarted","Data":"6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018"}
Jul 27 20:22:08.716132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.716099    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6" event={"ID":"d339a66c-c034-46c1-ae27-cc4110066d0d","Type":"ContainerStarted","Data":"c3b49f986358c0388225d726d33f3459bdecf72d87dd144d46be76db0ecf9705"}
Jul 27 20:22:08.716646 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.716142    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:08.717550 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.717512    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" event={"ID":"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b","Type":"ContainerStarted","Data":"5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79"}
Jul 27 20:22:08.717670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.717579    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:08.717670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.717603    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" event={"ID":"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b","Type":"ContainerStarted","Data":"912ce7c902744a9da5582d869179071f6013fd3c44d17fbd93e65148fa0eea51"}
Jul 27 20:22:08.718930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.718908    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk" event={"ID":"a339ace1-88ea-441f-8306-daad94f35ec4","Type":"ContainerStarted","Data":"d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56"}
Jul 27 20:22:08.719017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.718950    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:08.719017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.718967    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk" event={"ID":"a339ace1-88ea-441f-8306-daad94f35ec4","Type":"ContainerStarted","Data":"22b2ce2aa35c6f52e2f5d53ff47b3fafb85d9e0d59821a843c2b894eb82a4bae"}
Jul 27 20:22:08.738266 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.738222    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6" podStartSLOduration=1.738207025 podStartE2EDuration="1.738207025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:08.737013273 +0000 UTC m=+3477.732861252" watchObservedRunningTime="2026-07-27 20:22:08.738207025 +0000 UTC m=+3477.734055003"
Jul 27 20:22:08.784454 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.784399    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" podStartSLOduration=2.209935114 podStartE2EDuration="2.784383551s" totalImagesPullingTime="574.448437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:08.766001039 +0000 UTC m=+3477.761849018" watchObservedRunningTime="2026-07-27 20:22:08.784383551 +0000 UTC m=+3477.780231579"
Jul 27 20:22:08.784620 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:08.784483    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk" podStartSLOduration=1.784478215 podStartE2EDuration="1.784478215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:08.78238525 +0000 UTC m=+3477.778233227" watchObservedRunningTime="2026-07-27 20:22:08.784478215 +0000 UTC m=+3477.780326193"
Jul 27 20:22:11.110784 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:11.110749    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:11.673016 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:11.672990    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:22:11.673210 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:11.673160    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:22:11.969646 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:11.969556    2568 scope.go:117] "RemoveContainer" containerID="6f4a3658e754ba53881ec7ef6e2fa3c88c45bb08c3377f8b3cb760e7a241a696"
Jul 27 20:22:14.729081 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:14.729054    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:22:18.485220 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:18.485188    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:19.725688 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:19.725658    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:22:19.726067 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:19.725731    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:22:21.496212 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:21.496178    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:48.495024 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:48.494987    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:48.595641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:48.595597    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:53.956087 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.956035    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"]
Jul 27 20:22:53.959556 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.959539    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:53.962549 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.962514    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-43d589\"/\"default-dockercfg-8mh6d\""
Jul 27 20:22:53.963799 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.963781    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-43d589\"/\"openshift-service-ca.crt\""
Jul 27 20:22:53.963881 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.963803    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-43d589\"/\"kube-root-ca.crt\""
Jul 27 20:22:53.963881 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.963805    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-43d589\"/\"auth-test-model-43d589-kserve-self-signed-certs\""
Jul 27 20:22:53.970634 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:53.970611    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"]
Jul 27 20:22:54.031900 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.031860    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-model-cache\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.032057 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.031909    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-home\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.032057 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.031973    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzdn\" (UniqueName: \"kubernetes.io/projected/eeb70d4a-e0b9-400b-975e-5c1541f096f2-kube-api-access-rdzdn\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.032057 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.032024    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-dshm\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.032057 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.032045    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tls-certs\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.032195 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.032069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tmp-dir\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.050828 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.050793    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:22:54.132930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.132891    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tls-certs\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.132930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.132931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tmp-dir\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133169 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133019    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-model-cache\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133169 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133068    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-home\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133169 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133111    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzdn\" (UniqueName: \"kubernetes.io/projected/eeb70d4a-e0b9-400b-975e-5c1541f096f2-kube-api-access-rdzdn\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133335 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133262    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-dshm\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133336    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tmp-dir\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133393 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133365    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-model-cache\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.133500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.133410    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-home\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.135468 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.135444    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-dshm\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.135729 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.135714    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tls-certs\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.142791 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.142767    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzdn\" (UniqueName: \"kubernetes.io/projected/eeb70d4a-e0b9-400b-975e-5c1541f096f2-kube-api-access-rdzdn\") pod \"auth-test-model-43d589-kserve-5b585dd87f-s5skq\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") " pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.269422 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.269387    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:22:54.405619 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.405597    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"]
Jul 27 20:22:54.407585 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:22:54.407551    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeb70d4a_e0b9_400b_975e_5c1541f096f2.slice/crio-f2fab64afce6343f3ad8dcfcb4f432763d9aff9bb5e14517c60f03cfb90bb2f9 WatchSource:0}: Error finding container f2fab64afce6343f3ad8dcfcb4f432763d9aff9bb5e14517c60f03cfb90bb2f9: Status 404 returned error can't find the container with id f2fab64afce6343f3ad8dcfcb4f432763d9aff9bb5e14517c60f03cfb90bb2f9
Jul 27 20:22:54.945132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.945095    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq" event={"ID":"eeb70d4a-e0b9-400b-975e-5c1541f096f2","Type":"ContainerStarted","Data":"153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488"}
Jul 27 20:22:54.945132 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.945134    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq" event={"ID":"eeb70d4a-e0b9-400b-975e-5c1541f096f2","Type":"ContainerStarted","Data":"f2fab64afce6343f3ad8dcfcb4f432763d9aff9bb5e14517c60f03cfb90bb2f9"}
Jul 27 20:22:54.967482 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:22:54.967426    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq" podStartSLOduration=1.967411066 podStartE2EDuration="1.967411066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:22:54.966696671 +0000 UTC m=+3523.962544652" watchObservedRunningTime="2026-07-27 20:22:54.967411066 +0000 UTC m=+3523.963259045"
Jul 27 20:23:04.270020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:23:04.269972    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:23:04.270020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:23:04.270029    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:23:04.282833 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:23:04.282802    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:23:05.007283 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:23:05.007251    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:24:11.714554 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:24:11.714509    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:24:11.721912 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:24:11.721887    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:27:06.597574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:06.597509    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:06.687590 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:06.687560    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:08.451451 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:08.451409    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:08.552328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:08.552280    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:12.041285 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:12.041251    2568 scope.go:117] "RemoveContainer" containerID="51a91d0de14149839300a5f5d4eed2ac92ae8cb0eea518b8fd8496db3ef1f81b"
Jul 27 20:27:19.423540 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:19.423491    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:27.168782 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:27.168749    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:28.030226 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:28.030193    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"]
Jul 27 20:27:28.070450 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:28.070419    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:42.444137 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.444101    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"]
Jul 27 20:27:42.448540 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.448510    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.453392 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.453367    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"kube-root-ca.crt\""
Jul 27 20:27:42.453724 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.453669    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"openshift-service-ca.crt\""
Jul 27 20:27:42.453869 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.453843    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"default-dockercfg-qgzwl\""
Jul 27 20:27:42.454083 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.454058    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"sub-test-model-a78f44-kserve-self-signed-certs\""
Jul 27 20:27:42.457645 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.457623    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"]
Jul 27 20:27:42.466316 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.466293    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:42.485195 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.485172    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-dshm\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.485322 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.485215    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30aac9f6-6c88-4666-8985-52fda19d4222-tls-certs\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.485322 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.485242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-model-cache\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.485399 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.485323    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8tw\" (UniqueName: \"kubernetes.io/projected/30aac9f6-6c88-4666-8985-52fda19d4222-kube-api-access-tw8tw\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.485399 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.485362    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-tmp-dir\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.485399 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.485382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-home\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586239 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586212    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-tmp-dir\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586246    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-home\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586287    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-dshm\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586326    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30aac9f6-6c88-4666-8985-52fda19d4222-tls-certs\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-model-cache\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586394 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586389    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8tw\" (UniqueName: \"kubernetes.io/projected/30aac9f6-6c88-4666-8985-52fda19d4222-kube-api-access-tw8tw\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586737 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586706    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-tmp-dir\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586737 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586729    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-home\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.586900 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.586873    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-model-cache\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.588737 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.588701    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-dshm\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.589042 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.589023    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30aac9f6-6c88-4666-8985-52fda19d4222-tls-certs\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.602156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.602131    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8tw\" (UniqueName: \"kubernetes.io/projected/30aac9f6-6c88-4666-8985-52fda19d4222-kube-api-access-tw8tw\") pod \"sub-test-model-a78f44-kserve-86d959856c-cmkzx\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") " pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.760464 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.760364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:42.898624 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:42.898596    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"]
Jul 27 20:27:42.901166 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:27:42.901133    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30aac9f6_6c88_4666_8985_52fda19d4222.slice/crio-56b300718aeff8920b0ec824a5850aa1d98cc4c9a4b21a7756627cbbe25b28a8 WatchSource:0}: Error finding container 56b300718aeff8920b0ec824a5850aa1d98cc4c9a4b21a7756627cbbe25b28a8: Status 404 returned error can't find the container with id 56b300718aeff8920b0ec824a5850aa1d98cc4c9a4b21a7756627cbbe25b28a8
Jul 27 20:27:43.051508 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.051445    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq" podUID="eeb70d4a-e0b9-400b-975e-5c1541f096f2" containerName="main" containerID="cri-o://153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488" gracePeriod=45
Jul 27 20:27:43.291063 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.291036    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:27:43.393447 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393352    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tmp-dir\") pod \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") "
Jul 27 20:27:43.393447 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393388    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tls-certs\") pod \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") "
Jul 27 20:27:43.393447 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393419    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-home\") pod \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") "
Jul 27 20:27:43.393762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393509    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdzdn\" (UniqueName: \"kubernetes.io/projected/eeb70d4a-e0b9-400b-975e-5c1541f096f2-kube-api-access-rdzdn\") pod \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") "
Jul 27 20:27:43.393762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393611    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-dshm\") pod \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") "
Jul 27 20:27:43.393762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393638    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-model-cache\") pod \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\" (UID: \"eeb70d4a-e0b9-400b-975e-5c1541f096f2\") "
Jul 27 20:27:43.393762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393721    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeb70d4a-e0b9-400b-975e-5c1541f096f2" (UID: "eeb70d4a-e0b9-400b-975e-5c1541f096f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:27:43.393949 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393758    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-home" (OuterVolumeSpecName: "home") pod "eeb70d4a-e0b9-400b-975e-5c1541f096f2" (UID: "eeb70d4a-e0b9-400b-975e-5c1541f096f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:27:43.394004 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.393943    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeb70d4a-e0b9-400b-975e-5c1541f096f2" (UID: "eeb70d4a-e0b9-400b-975e-5c1541f096f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:27:43.394058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.394003    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:27:43.394058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.394016    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:27:43.395876 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.395855    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeb70d4a-e0b9-400b-975e-5c1541f096f2" (UID: "eeb70d4a-e0b9-400b-975e-5c1541f096f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:27:43.395983 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.395899    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-dshm" (OuterVolumeSpecName: "dshm") pod "eeb70d4a-e0b9-400b-975e-5c1541f096f2" (UID: "eeb70d4a-e0b9-400b-975e-5c1541f096f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:27:43.396029 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.396002    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb70d4a-e0b9-400b-975e-5c1541f096f2-kube-api-access-rdzdn" (OuterVolumeSpecName: "kube-api-access-rdzdn") pod "eeb70d4a-e0b9-400b-975e-5c1541f096f2" (UID: "eeb70d4a-e0b9-400b-975e-5c1541f096f2"). InnerVolumeSpecName "kube-api-access-rdzdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:27:43.398680 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.398656    2568 generic.go:356] "Generic (PLEG): container finished" podID="eeb70d4a-e0b9-400b-975e-5c1541f096f2" containerID="153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488" exitCode=0
Jul 27 20:27:43.398768 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.398699    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"
Jul 27 20:27:43.398768 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.398736    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq" event={"ID":"eeb70d4a-e0b9-400b-975e-5c1541f096f2","Type":"ContainerDied","Data":"153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488"}
Jul 27 20:27:43.398843 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.398778    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq" event={"ID":"eeb70d4a-e0b9-400b-975e-5c1541f096f2","Type":"ContainerDied","Data":"f2fab64afce6343f3ad8dcfcb4f432763d9aff9bb5e14517c60f03cfb90bb2f9"}
Jul 27 20:27:43.398843 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.398801    2568 scope.go:117] "RemoveContainer" containerID="153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488"
Jul 27 20:27:43.400460 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.400438    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx" event={"ID":"30aac9f6-6c88-4666-8985-52fda19d4222","Type":"ContainerStarted","Data":"c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128"}
Jul 27 20:27:43.400613 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.400473    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx" event={"ID":"30aac9f6-6c88-4666-8985-52fda19d4222","Type":"ContainerStarted","Data":"56b300718aeff8920b0ec824a5850aa1d98cc4c9a4b21a7756627cbbe25b28a8"}
Jul 27 20:27:43.409680 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.409664    2568 scope.go:117] "RemoveContainer" containerID="153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488"
Jul 27 20:27:43.409940 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:27:43.409922    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488\": container with ID starting with 153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488 not found: ID does not exist" containerID="153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488"
Jul 27 20:27:43.409994 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.409949    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488"} err="failed to get container status \"153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488\": rpc error: code = NotFound desc = could not find container \"153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488\": container with ID starting with 153db3f4fc5dce097f5757401a2c055492b6a4ca5daf03ec77c5fd038ef4f488 not found: ID does not exist"
Jul 27 20:27:43.420301 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.420262    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx" podStartSLOduration=1.420247829 podStartE2EDuration="1.420247829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:27:43.419653681 +0000 UTC m=+3812.415501659" watchObservedRunningTime="2026-07-27 20:27:43.420247829 +0000 UTC m=+3812.416095807"
Jul 27 20:27:43.436478 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.436455    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"]
Jul 27 20:27:43.443040 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.443013    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-43d589/auth-test-model-43d589-kserve-5b585dd87f-s5skq"]
Jul 27 20:27:43.495359 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.495338    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzdn\" (UniqueName: \"kubernetes.io/projected/eeb70d4a-e0b9-400b-975e-5c1541f096f2-kube-api-access-rdzdn\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:27:43.495359 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.495358    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:27:43.495733 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.495371    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb70d4a-e0b9-400b-975e-5c1541f096f2-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:27:43.495733 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.495380    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb70d4a-e0b9-400b-975e-5c1541f096f2-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:27:43.568402 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:43.568323    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb70d4a-e0b9-400b-975e-5c1541f096f2" path="/var/lib/kubelet/pods/eeb70d4a-e0b9-400b-975e-5c1541f096f2/volumes"
Jul 27 20:27:52.013078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:52.013041    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:27:52.761328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:52.761290    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:52.761517 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:52.761347    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:52.774604 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:52.774581    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:27:53.461904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:27:53.461875    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:28:04.878099 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:28:04.878062    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:28:12.061172 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:28:12.061143    2568 scope.go:117] "RemoveContainer" containerID="dc95a7a0b8825bb096af997c43e5fb3a7cdd35662a502ad76a0297f6190712dd"
Jul 27 20:29:02.378574 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:02.378517    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:29:02.479137 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:02.479106    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:29:11.748228 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:11.748196    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:29:11.761503 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:11.761473    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:29:11.765498 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:11.765467    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"]
Jul 27 20:29:11.775071 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:11.775032    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:29:12.693775 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:12.693741    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:29:26.789643 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:26.789604    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx" podUID="30aac9f6-6c88-4666-8985-52fda19d4222" containerName="main" containerID="cri-o://c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128" gracePeriod=45
Jul 27 20:29:27.039770 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.039710    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:29:27.176710 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.176682    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-tmp-dir\") pod \"30aac9f6-6c88-4666-8985-52fda19d4222\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") "
Jul 27 20:29:27.176886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.176733    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30aac9f6-6c88-4666-8985-52fda19d4222-tls-certs\") pod \"30aac9f6-6c88-4666-8985-52fda19d4222\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") "
Jul 27 20:29:27.176886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.176766    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-home\") pod \"30aac9f6-6c88-4666-8985-52fda19d4222\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") "
Jul 27 20:29:27.176886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.176798    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-model-cache\") pod \"30aac9f6-6c88-4666-8985-52fda19d4222\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") "
Jul 27 20:29:27.176886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.176845    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-dshm\") pod \"30aac9f6-6c88-4666-8985-52fda19d4222\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") "
Jul 27 20:29:27.176886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.176873    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw8tw\" (UniqueName: \"kubernetes.io/projected/30aac9f6-6c88-4666-8985-52fda19d4222-kube-api-access-tw8tw\") pod \"30aac9f6-6c88-4666-8985-52fda19d4222\" (UID: \"30aac9f6-6c88-4666-8985-52fda19d4222\") "
Jul 27 20:29:27.177133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.177015    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-home" (OuterVolumeSpecName: "home") pod "30aac9f6-6c88-4666-8985-52fda19d4222" (UID: "30aac9f6-6c88-4666-8985-52fda19d4222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:29:27.177133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.177026    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30aac9f6-6c88-4666-8985-52fda19d4222" (UID: "30aac9f6-6c88-4666-8985-52fda19d4222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:29:27.177246 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.177155    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-model-cache" (OuterVolumeSpecName: "model-cache") pod "30aac9f6-6c88-4666-8985-52fda19d4222" (UID: "30aac9f6-6c88-4666-8985-52fda19d4222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:29:27.177303 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.177257    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:29:27.177303 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.177276    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:29:27.177303 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.177285    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:29:27.179138 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.179110    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-dshm" (OuterVolumeSpecName: "dshm") pod "30aac9f6-6c88-4666-8985-52fda19d4222" (UID: "30aac9f6-6c88-4666-8985-52fda19d4222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:29:27.179234 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.179169    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30aac9f6-6c88-4666-8985-52fda19d4222-kube-api-access-tw8tw" (OuterVolumeSpecName: "kube-api-access-tw8tw") pod "30aac9f6-6c88-4666-8985-52fda19d4222" (UID: "30aac9f6-6c88-4666-8985-52fda19d4222"). InnerVolumeSpecName "kube-api-access-tw8tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:29:27.179234 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.179216    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30aac9f6-6c88-4666-8985-52fda19d4222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30aac9f6-6c88-4666-8985-52fda19d4222" (UID: "30aac9f6-6c88-4666-8985-52fda19d4222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:29:27.277794 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.277747    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30aac9f6-6c88-4666-8985-52fda19d4222-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:29:27.277794 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.277787    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30aac9f6-6c88-4666-8985-52fda19d4222-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:29:27.277794 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.277797    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8tw\" (UniqueName: \"kubernetes.io/projected/30aac9f6-6c88-4666-8985-52fda19d4222-kube-api-access-tw8tw\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:29:27.903713 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.903631    2568 generic.go:356] "Generic (PLEG): container finished" podID="30aac9f6-6c88-4666-8985-52fda19d4222" containerID="c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128" exitCode=0
Jul 27 20:29:27.903713 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.903673    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"
Jul 27 20:29:27.904156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.903709    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx" event={"ID":"30aac9f6-6c88-4666-8985-52fda19d4222","Type":"ContainerDied","Data":"c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128"}
Jul 27 20:29:27.904156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.903754    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx" event={"ID":"30aac9f6-6c88-4666-8985-52fda19d4222","Type":"ContainerDied","Data":"56b300718aeff8920b0ec824a5850aa1d98cc4c9a4b21a7756627cbbe25b28a8"}
Jul 27 20:29:27.904156 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.903771    2568 scope.go:117] "RemoveContainer" containerID="c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128"
Jul 27 20:29:27.912852 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.912835    2568 scope.go:117] "RemoveContainer" containerID="c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128"
Jul 27 20:29:27.913105 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:29:27.913086    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128\": container with ID starting with c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128 not found: ID does not exist" containerID="c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128"
Jul 27 20:29:27.913182 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.913113    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128"} err="failed to get container status \"c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128\": rpc error: code = NotFound desc = could not find container \"c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128\": container with ID starting with c46e690542b01a3d1d39d73b0086c17111b9b93ec5b180863ad854e67eab3128 not found: ID does not exist"
Jul 27 20:29:27.932175 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.932147    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"]
Jul 27 20:29:27.937199 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:27.937172    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/sub-test-model-a78f44-kserve-86d959856c-cmkzx"]
Jul 27 20:29:29.568107 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:29.568075    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30aac9f6-6c88-4666-8985-52fda19d4222" path="/var/lib/kubelet/pods/30aac9f6-6c88-4666-8985-52fda19d4222/volumes"
Jul 27 20:29:30.247543 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.247497    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"]
Jul 27 20:29:30.248157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.248084    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb70d4a-e0b9-400b-975e-5c1541f096f2" containerName="main"
Jul 27 20:29:30.248157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.248108    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb70d4a-e0b9-400b-975e-5c1541f096f2" containerName="main"
Jul 27 20:29:30.248157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.248145    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30aac9f6-6c88-4666-8985-52fda19d4222" containerName="main"
Jul 27 20:29:30.248157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.248152    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="30aac9f6-6c88-4666-8985-52fda19d4222" containerName="main"
Jul 27 20:29:30.248452 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.248245    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="30aac9f6-6c88-4666-8985-52fda19d4222" containerName="main"
Jul 27 20:29:30.248452 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.248259    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb70d4a-e0b9-400b-975e-5c1541f096f2" containerName="main"
Jul 27 20:29:30.253116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.253091    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.260091 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.260066    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"kube-root-ca.crt\""
Jul 27 20:29:30.260337 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.260317    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"rate-test-model-a78f44-kserve-self-signed-certs\""
Jul 27 20:29:30.260562 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.260544    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"default-dockercfg-qgzwl\""
Jul 27 20:29:30.261345 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.260763    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a78f44\"/\"openshift-service-ca.crt\""
Jul 27 20:29:30.263208 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.263185    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"]
Jul 27 20:29:30.286699 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.286669    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:29:30.304781 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.304752    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e0944-10e8-47da-9e53-bcca93eddb90-tls-certs\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.304923 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.304815    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-tmp-dir\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.304923 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.304879    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-dshm\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.304923 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.304917    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-home\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.305051 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.304935    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-model-cache\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.305051 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.304950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8wfb\" (UniqueName: \"kubernetes.io/projected/9f2e0944-10e8-47da-9e53-bcca93eddb90-kube-api-access-f8wfb\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.405794 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.405759    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-dshm\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.405982 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.405808    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-home\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.405982 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.405827    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-model-cache\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.405982 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.405850    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8wfb\" (UniqueName: \"kubernetes.io/projected/9f2e0944-10e8-47da-9e53-bcca93eddb90-kube-api-access-f8wfb\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.405982 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.405906    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e0944-10e8-47da-9e53-bcca93eddb90-tls-certs\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.405982 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.405941    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-tmp-dir\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.406272 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.406251    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-model-cache\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.406329 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.406295    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-home\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.406375 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.406339    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-tmp-dir\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.408259 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.408239    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-dshm\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.408516 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.408501    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e0944-10e8-47da-9e53-bcca93eddb90-tls-certs\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.417861 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.417837    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8wfb\" (UniqueName: \"kubernetes.io/projected/9f2e0944-10e8-47da-9e53-bcca93eddb90-kube-api-access-f8wfb\") pod \"rate-test-model-a78f44-kserve-677bbb9d67-nnczv\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") " pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.564387 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.564355    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:30.705167 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.705127    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"]
Jul 27 20:29:30.707442 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:29:30.707418    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f2e0944_10e8_47da_9e53_bcca93eddb90.slice/crio-0496bc6485bf810d44a99e8b5256e1189d7d755dbfc4cea5dbca2f52548a518e WatchSource:0}: Error finding container 0496bc6485bf810d44a99e8b5256e1189d7d755dbfc4cea5dbca2f52548a518e: Status 404 returned error can't find the container with id 0496bc6485bf810d44a99e8b5256e1189d7d755dbfc4cea5dbca2f52548a518e
Jul 27 20:29:30.921305 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.921216    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv" event={"ID":"9f2e0944-10e8-47da-9e53-bcca93eddb90","Type":"ContainerStarted","Data":"4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc"}
Jul 27 20:29:30.921305 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.921255    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv" event={"ID":"9f2e0944-10e8-47da-9e53-bcca93eddb90","Type":"ContainerStarted","Data":"0496bc6485bf810d44a99e8b5256e1189d7d755dbfc4cea5dbca2f52548a518e"}
Jul 27 20:29:30.941458 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:30.941407    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv" podStartSLOduration=0.941388014 podStartE2EDuration="941.388014ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:29:30.939592985 +0000 UTC m=+3919.935440955" watchObservedRunningTime="2026-07-27 20:29:30.941388014 +0000 UTC m=+3919.937236086"
Jul 27 20:29:32.893912 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:32.893878    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:29:40.565330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:40.565295    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:40.565330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:40.565335    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:40.577944 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:40.577919    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:40.981865 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:40.981787    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:29:52.270715 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:29:52.270688    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:30:00.134549 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.134487    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"]
Jul 27 20:30:00.137888 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.137872    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"]
Jul 27 20:30:00.137998 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.137980    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"
Jul 27 20:30:00.140500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.140481    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:30:00.141125 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.141109    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"
Jul 27 20:30:00.149757 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.149734    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"]
Jul 27 20:30:00.151711 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.151687    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"
Jul 27 20:30:00.152802 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.152780    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"]
Jul 27 20:30:00.159069 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.159050    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"
Jul 27 20:30:00.301234 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.301203    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"]
Jul 27 20:30:00.309615 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:30:00.309581    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda323b8e3_0cf3_4985_8a48_779815d2856b.slice/crio-ae7ccccc07de4ad4d6d04c5177e0348fe3810d758415015557c49aea2f361f39 WatchSource:0}: Error finding container ae7ccccc07de4ad4d6d04c5177e0348fe3810d758415015557c49aea2f361f39: Status 404 returned error can't find the container with id ae7ccccc07de4ad4d6d04c5177e0348fe3810d758415015557c49aea2f361f39
Jul 27 20:30:00.548020 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:00.547930    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"]
Jul 27 20:30:00.550167 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:30:00.550139    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4575446_4609_4b9a_ba93_057e93e5b2ec.slice/crio-9dc4157c7536c8678b23d6719e496300093fae29dc173962ceed9a562b7a7800 WatchSource:0}: Error finding container 9dc4157c7536c8678b23d6719e496300093fae29dc173962ceed9a562b7a7800: Status 404 returned error can't find the container with id 9dc4157c7536c8678b23d6719e496300093fae29dc173962ceed9a562b7a7800
Jul 27 20:30:01.070767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:01.070657    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn" event={"ID":"b4575446-4609-4b9a-ba93-057e93e5b2ec","Type":"ContainerStarted","Data":"e5f0b91e7df3cb5191580146a274153ccacacadd15c99b60f626fdd339b9139b"}
Jul 27 20:30:01.070767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:01.070764    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn" event={"ID":"b4575446-4609-4b9a-ba93-057e93e5b2ec","Type":"ContainerStarted","Data":"9dc4157c7536c8678b23d6719e496300093fae29dc173962ceed9a562b7a7800"}
Jul 27 20:30:01.072540 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:01.072489    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44" event={"ID":"a323b8e3-0cf3-4985-8a48-779815d2856b","Type":"ContainerStarted","Data":"4d6609d3bbb8fc9f7211cd249a6287aa450e7b3b97892551339a424af448ff08"}
Jul 27 20:30:01.072679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:01.072594    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44" event={"ID":"a323b8e3-0cf3-4985-8a48-779815d2856b","Type":"ContainerStarted","Data":"ae7ccccc07de4ad4d6d04c5177e0348fe3810d758415015557c49aea2f361f39"}
Jul 27 20:30:01.087196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:01.087147    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn" podStartSLOduration=1.087132085 podStartE2EDuration="1.087132085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:01.085424877 +0000 UTC m=+3950.081272857" watchObservedRunningTime="2026-07-27 20:30:01.087132085 +0000 UTC m=+3950.082980100"
Jul 27 20:30:01.100775 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:01.100728    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44" podStartSLOduration=1.100704678 podStartE2EDuration="1.100704678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:30:01.100457604 +0000 UTC m=+3950.096305583" watchObservedRunningTime="2026-07-27 20:30:01.100704678 +0000 UTC m=+3950.096552683"
Jul 27 20:30:02.079384 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:02.079304    2568 generic.go:356] "Generic (PLEG): container finished" podID="b4575446-4609-4b9a-ba93-057e93e5b2ec" containerID="e5f0b91e7df3cb5191580146a274153ccacacadd15c99b60f626fdd339b9139b" exitCode=0
Jul 27 20:30:02.079807 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:02.079387    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn" event={"ID":"b4575446-4609-4b9a-ba93-057e93e5b2ec","Type":"ContainerDied","Data":"e5f0b91e7df3cb5191580146a274153ccacacadd15c99b60f626fdd339b9139b"}
Jul 27 20:30:02.080997 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:02.080963    2568 generic.go:356] "Generic (PLEG): container finished" podID="a323b8e3-0cf3-4985-8a48-779815d2856b" containerID="4d6609d3bbb8fc9f7211cd249a6287aa450e7b3b97892551339a424af448ff08" exitCode=0
Jul 27 20:30:02.081090 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:02.080994    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44" event={"ID":"a323b8e3-0cf3-4985-8a48-779815d2856b","Type":"ContainerDied","Data":"4d6609d3bbb8fc9f7211cd249a6287aa450e7b3b97892551339a424af448ff08"}
Jul 27 20:30:03.230362 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:03.230340    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"
Jul 27 20:30:03.256341 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:03.256316    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"
Jul 27 20:30:04.092114 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:04.092085    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44"
Jul 27 20:30:04.092114 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:04.092118    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753070-9pd44" event={"ID":"a323b8e3-0cf3-4985-8a48-779815d2856b","Type":"ContainerDied","Data":"ae7ccccc07de4ad4d6d04c5177e0348fe3810d758415015557c49aea2f361f39"}
Jul 27 20:30:04.092354 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:04.092161    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7ccccc07de4ad4d6d04c5177e0348fe3810d758415015557c49aea2f361f39"
Jul 27 20:30:04.093648 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:04.093623    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn" event={"ID":"b4575446-4609-4b9a-ba93-057e93e5b2ec","Type":"ContainerDied","Data":"9dc4157c7536c8678b23d6719e496300093fae29dc173962ceed9a562b7a7800"}
Jul 27 20:30:04.093764 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:04.093651    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"
Jul 27 20:30:04.093764 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:30:04.093663    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dc4157c7536c8678b23d6719e496300093fae29dc173962ceed9a562b7a7800"
Jul 27 20:31:06.042146 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:06.042112    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:18.252930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:18.252897    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"]
Jul 27 20:31:18.265311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:18.265254    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:19.142409 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:19.142375    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:32.113290 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113213    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7350-140181158098816"]
Jul 27 20:31:32.113665 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113654    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a323b8e3-0cf3-4985-8a48-779815d2856b" containerName="cleanup"
Jul 27 20:31:32.113714 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113667    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a323b8e3-0cf3-4985-8a48-779815d2856b" containerName="cleanup"
Jul 27 20:31:32.113714 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113692    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4575446-4609-4b9a-ba93-057e93e5b2ec" containerName="cleanup"
Jul 27 20:31:32.113714 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113703    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4575446-4609-4b9a-ba93-057e93e5b2ec" containerName="cleanup"
Jul 27 20:31:32.113809 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113765    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a323b8e3-0cf3-4985-8a48-779815d2856b" containerName="cleanup"
Jul 27 20:31:32.113809 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.113780    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4575446-4609-4b9a-ba93-057e93e5b2ec" containerName="cleanup"
Jul 27 20:31:32.116923 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.116907    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:32.119733 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.119714    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2wsq4\""
Jul 27 20:31:32.124828 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.124088    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-140181158098816"]
Jul 27 20:31:32.241463 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.241420    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/46d3e434-3587-4ef4-be4b-562c98073ac4-kube-api-access-kk5ls\") pod \"test-curl-7350-140181158098816\" (UID: \"46d3e434-3587-4ef4-be4b-562c98073ac4\") " pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:32.342125 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.342091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/46d3e434-3587-4ef4-be4b-562c98073ac4-kube-api-access-kk5ls\") pod \"test-curl-7350-140181158098816\" (UID: \"46d3e434-3587-4ef4-be4b-562c98073ac4\") " pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:32.350975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.350951    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/46d3e434-3587-4ef4-be4b-562c98073ac4-kube-api-access-kk5ls\") pod \"test-curl-7350-140181158098816\" (UID: \"46d3e434-3587-4ef4-be4b-562c98073ac4\") " pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:32.427849 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.427769    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:32.569927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.569902    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7350-140181158098816"]
Jul 27 20:31:32.572289 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:31:32.572262    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46d3e434_3587_4ef4_be4b_562c98073ac4.slice/crio-ae3be4f5df080c19ddc7fdf63453f2091d34bf6d80c3cf07aa54f2864d975c90 WatchSource:0}: Error finding container ae3be4f5df080c19ddc7fdf63453f2091d34bf6d80c3cf07aa54f2864d975c90: Status 404 returned error can't find the container with id ae3be4f5df080c19ddc7fdf63453f2091d34bf6d80c3cf07aa54f2864d975c90
Jul 27 20:31:32.573632 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:32.573612    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:31:33.272567 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.272510    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv" podUID="9f2e0944-10e8-47da-9e53-bcca93eddb90" containerName="main" containerID="cri-o://4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc" gracePeriod=45
Jul 27 20:31:33.541948 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.541923    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:31:33.551755 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.551724    2568 generic.go:356] "Generic (PLEG): container finished" podID="9f2e0944-10e8-47da-9e53-bcca93eddb90" containerID="4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc" exitCode=0
Jul 27 20:31:33.551927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.551769    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"
Jul 27 20:31:33.551927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.551817    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv" event={"ID":"9f2e0944-10e8-47da-9e53-bcca93eddb90","Type":"ContainerDied","Data":"4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc"}
Jul 27 20:31:33.551927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.551871    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv" event={"ID":"9f2e0944-10e8-47da-9e53-bcca93eddb90","Type":"ContainerDied","Data":"0496bc6485bf810d44a99e8b5256e1189d7d755dbfc4cea5dbca2f52548a518e"}
Jul 27 20:31:33.551927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.551896    2568 scope.go:117] "RemoveContainer" containerID="4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc"
Jul 27 20:31:33.553487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.553463    2568 generic.go:356] "Generic (PLEG): container finished" podID="46d3e434-3587-4ef4-be4b-562c98073ac4" containerID="e7d51178ab7579dd71ea2c153cb9800eac7026e7f73027dc10f01ee3b8785b5c" exitCode=0
Jul 27 20:31:33.553673 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.553648    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-140181158098816" event={"ID":"46d3e434-3587-4ef4-be4b-562c98073ac4","Type":"ContainerDied","Data":"e7d51178ab7579dd71ea2c153cb9800eac7026e7f73027dc10f01ee3b8785b5c"}
Jul 27 20:31:33.553728 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.553695    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7350-140181158098816" event={"ID":"46d3e434-3587-4ef4-be4b-562c98073ac4","Type":"ContainerStarted","Data":"ae3be4f5df080c19ddc7fdf63453f2091d34bf6d80c3cf07aa54f2864d975c90"}
Jul 27 20:31:33.561576 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.561550    2568 scope.go:117] "RemoveContainer" containerID="4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc"
Jul 27 20:31:33.561947 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:31:33.561917    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc\": container with ID starting with 4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc not found: ID does not exist" containerID="4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc"
Jul 27 20:31:33.562036 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.561958    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc"} err="failed to get container status \"4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc\": rpc error: code = NotFound desc = could not find container \"4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc\": container with ID starting with 4e68b0f834d75ab68edcd9ebb706cd67c6191189886cf37aba337892b77266cc not found: ID does not exist"
Jul 27 20:31:33.655330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655297    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e0944-10e8-47da-9e53-bcca93eddb90-tls-certs\") pod \"9f2e0944-10e8-47da-9e53-bcca93eddb90\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") "
Jul 27 20:31:33.655330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655331    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-home\") pod \"9f2e0944-10e8-47da-9e53-bcca93eddb90\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") "
Jul 27 20:31:33.655639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655354    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8wfb\" (UniqueName: \"kubernetes.io/projected/9f2e0944-10e8-47da-9e53-bcca93eddb90-kube-api-access-f8wfb\") pod \"9f2e0944-10e8-47da-9e53-bcca93eddb90\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") "
Jul 27 20:31:33.655639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655424    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-dshm\") pod \"9f2e0944-10e8-47da-9e53-bcca93eddb90\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") "
Jul 27 20:31:33.655639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655479    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-tmp-dir\") pod \"9f2e0944-10e8-47da-9e53-bcca93eddb90\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") "
Jul 27 20:31:33.655639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655509    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-model-cache\") pod \"9f2e0944-10e8-47da-9e53-bcca93eddb90\" (UID: \"9f2e0944-10e8-47da-9e53-bcca93eddb90\") "
Jul 27 20:31:33.655639 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655614    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-home" (OuterVolumeSpecName: "home") pod "9f2e0944-10e8-47da-9e53-bcca93eddb90" (UID: "9f2e0944-10e8-47da-9e53-bcca93eddb90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:31:33.655967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655808    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f2e0944-10e8-47da-9e53-bcca93eddb90" (UID: "9f2e0944-10e8-47da-9e53-bcca93eddb90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:31:33.655967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655844    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f2e0944-10e8-47da-9e53-bcca93eddb90" (UID: "9f2e0944-10e8-47da-9e53-bcca93eddb90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:31:33.655967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655929    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-tmp-dir\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:33.655967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655946    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-model-cache\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:33.655967 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.655958    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-home\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:33.657766 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.657737    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-dshm" (OuterVolumeSpecName: "dshm") pod "9f2e0944-10e8-47da-9e53-bcca93eddb90" (UID: "9f2e0944-10e8-47da-9e53-bcca93eddb90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:31:33.657859 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.657777    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2e0944-10e8-47da-9e53-bcca93eddb90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f2e0944-10e8-47da-9e53-bcca93eddb90" (UID: "9f2e0944-10e8-47da-9e53-bcca93eddb90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:31:33.657859 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.657807    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2e0944-10e8-47da-9e53-bcca93eddb90-kube-api-access-f8wfb" (OuterVolumeSpecName: "kube-api-access-f8wfb") pod "9f2e0944-10e8-47da-9e53-bcca93eddb90" (UID: "9f2e0944-10e8-47da-9e53-bcca93eddb90"). InnerVolumeSpecName "kube-api-access-f8wfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:31:33.757254 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.757215    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2e0944-10e8-47da-9e53-bcca93eddb90-tls-certs\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:33.757254 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.757246    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8wfb\" (UniqueName: \"kubernetes.io/projected/9f2e0944-10e8-47da-9e53-bcca93eddb90-kube-api-access-f8wfb\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:33.757254 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.757255    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2e0944-10e8-47da-9e53-bcca93eddb90-dshm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:33.875085 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.875053    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"]
Jul 27 20:31:33.879259 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:33.879235    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a78f44/rate-test-model-a78f44-kserve-677bbb9d67-nnczv"]
Jul 27 20:31:34.707836 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.707811    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:34.728799 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.728773    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7350-140181158098816_46d3e434-3587-4ef4-be4b-562c98073ac4/test-curl-7350-140181158098816/0.log"
Jul 27 20:31:34.752358 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.752321    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7350-140181158098816"]
Jul 27 20:31:34.754492 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.754468    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7350-140181158098816"]
Jul 27 20:31:34.867944 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.867856    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/46d3e434-3587-4ef4-be4b-562c98073ac4-kube-api-access-kk5ls\") pod \"46d3e434-3587-4ef4-be4b-562c98073ac4\" (UID: \"46d3e434-3587-4ef4-be4b-562c98073ac4\") "
Jul 27 20:31:34.870114 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.870076    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d3e434-3587-4ef4-be4b-562c98073ac4-kube-api-access-kk5ls" (OuterVolumeSpecName: "kube-api-access-kk5ls") pod "46d3e434-3587-4ef4-be4b-562c98073ac4" (UID: "46d3e434-3587-4ef4-be4b-562c98073ac4"). InnerVolumeSpecName "kube-api-access-kk5ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:31:34.969511 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:34.969473    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5ls\" (UniqueName: \"kubernetes.io/projected/46d3e434-3587-4ef4-be4b-562c98073ac4-kube-api-access-kk5ls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:31:35.567189 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:35.567161    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7350-140181158098816"
Jul 27 20:31:35.567713 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:35.567688    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d3e434-3587-4ef4-be4b-562c98073ac4" path="/var/lib/kubelet/pods/46d3e434-3587-4ef4-be4b-562c98073ac4/volumes"
Jul 27 20:31:35.568133 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:35.568114    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2e0944-10e8-47da-9e53-bcca93eddb90" path="/var/lib/kubelet/pods/9f2e0944-10e8-47da-9e53-bcca93eddb90/volumes"
Jul 27 20:31:35.568479 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:35.568465    2568 scope.go:117] "RemoveContainer" containerID="e7d51178ab7579dd71ea2c153cb9800eac7026e7f73027dc10f01ee3b8785b5c"
Jul 27 20:31:38.378840 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:38.378804    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:38.682997 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:38.682918    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:38.782677 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:38.782639    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:41.659931 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:41.659901    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:41.741572 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:41.741516    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:42.318971 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.318944    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"]
Jul 27 20:31:42.319351 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.319340    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2e0944-10e8-47da-9e53-bcca93eddb90" containerName="main"
Jul 27 20:31:42.319406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.319354    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2e0944-10e8-47da-9e53-bcca93eddb90" containerName="main"
Jul 27 20:31:42.319406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.319365    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d3e434-3587-4ef4-be4b-562c98073ac4" containerName="test-curl-7350-140181158098816"
Jul 27 20:31:42.319406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.319371    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d3e434-3587-4ef4-be4b-562c98073ac4" containerName="test-curl-7350-140181158098816"
Jul 27 20:31:42.319501 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.319427    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d3e434-3587-4ef4-be4b-562c98073ac4" containerName="test-curl-7350-140181158098816"
Jul 27 20:31:42.319501 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.319436    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2e0944-10e8-47da-9e53-bcca93eddb90" containerName="main"
Jul 27 20:31:42.323902 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.323873    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.326608 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.326588    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4ac78b-serving-cert\""
Jul 27 20:31:42.334070 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.334047    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"]
Jul 27 20:31:42.338507 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.338485    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b6435e-ff87-4882-b54d-01e331c82f4d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.338623 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.338594    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.338669 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.338645    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvt28\" (UniqueName: \"kubernetes.io/projected/68b6435e-ff87-4882-b54d-01e331c82f4d-kube-api-access-vvt28\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.347549 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.347511    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:42.439062 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.439032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.439246 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.439073    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvt28\" (UniqueName: \"kubernetes.io/projected/68b6435e-ff87-4882-b54d-01e331c82f4d-kube-api-access-vvt28\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.439246 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.439113    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b6435e-ff87-4882-b54d-01e331c82f4d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.439246 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:31:42.439173    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-serving-cert: secret "maas-api-e2e-ipp-a-4ac78b-serving-cert" not found
Jul 27 20:31:42.439246 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:31:42.439234    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls podName:68b6435e-ff87-4882-b54d-01e331c82f4d nodeName:}" failed. No retries permitted until 2026-07-27 20:31:42.939219262 +0000 UTC m=+4051.935067218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls") pod "maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" (UID: "68b6435e-ff87-4882-b54d-01e331c82f4d") : secret "maas-api-e2e-ipp-a-4ac78b-serving-cert" not found
Jul 27 20:31:42.439829 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.439813    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b6435e-ff87-4882-b54d-01e331c82f4d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.448661 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.448644    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvt28\" (UniqueName: \"kubernetes.io/projected/68b6435e-ff87-4882-b54d-01e331c82f4d-kube-api-access-vvt28\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.749577 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.749460    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:42.854307 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.854274    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:42.944188 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.944154    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.946762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.946731    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:42.952161 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.952137    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:42.975203 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.975170    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"]
Jul 27 20:31:42.978836 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.978821    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:42.981385 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.981366    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ac78b\""
Jul 27 20:31:42.981500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.981385    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ac78b-dockercfg-l4pjj\""
Jul 27 20:31:42.988582 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:42.988553    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"]
Jul 27 20:31:43.024345 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.024311    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"]
Jul 27 20:31:43.028074 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.028058    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.038222 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.038197    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"]
Jul 27 20:31:43.045376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.045345    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v4kj\" (UniqueName: \"kubernetes.io/projected/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-kube-api-access-8v4kj\") pod \"payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.045587 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.045569    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.146907 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.146870    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v4kj\" (UniqueName: \"kubernetes.io/projected/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-kube-api-access-8v4kj\") pod \"payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.146907 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.146915    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ppf\" (UniqueName: \"kubernetes.io/projected/5ac4051d-e079-4a5f-a191-c6bc5270aa47-kube-api-access-r4ppf\") pod \"payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.147163 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.146944    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac4051d-e079-4a5f-a191-c6bc5270aa47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.147163 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.147021    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.147716 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.147700    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.156244 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.156218    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v4kj\" (UniqueName: \"kubernetes.io/projected/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-kube-api-access-8v4kj\") pod \"payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.236681 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.236650    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:43.247547 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.247498    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac4051d-e079-4a5f-a191-c6bc5270aa47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.247693 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.247677    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ppf\" (UniqueName: \"kubernetes.io/projected/5ac4051d-e079-4a5f-a191-c6bc5270aa47-kube-api-access-r4ppf\") pod \"payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.248189 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.248160    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac4051d-e079-4a5f-a191-c6bc5270aa47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.256885 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.256856    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ppf\" (UniqueName: \"kubernetes.io/projected/5ac4051d-e079-4a5f-a191-c6bc5270aa47-kube-api-access-r4ppf\") pod \"payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.290614 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.290585    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.338767 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.338438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.388010 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.387968    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"]
Jul 27 20:31:43.459557 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.459286    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"]
Jul 27 20:31:43.464697 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:31:43.464662    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ae71d6_4f18_42de_a04a_b0a6fcd125e2.slice/crio-6be88922c785c7c456219f044424b64c9cd68e584466504fed178402e229c8d2 WatchSource:0}: Error finding container 6be88922c785c7c456219f044424b64c9cd68e584466504fed178402e229c8d2: Status 404 returned error can't find the container with id 6be88922c785c7c456219f044424b64c9cd68e584466504fed178402e229c8d2
Jul 27 20:31:43.512981 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.512955    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"]
Jul 27 20:31:43.515104 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:31:43.515081    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac4051d_e079_4a5f_a191_c6bc5270aa47.slice/crio-3b1025752cc7fdae4c6355d02587edb1cc5c96a361ddf515eeb9692cb6bb6015 WatchSource:0}: Error finding container 3b1025752cc7fdae4c6355d02587edb1cc5c96a361ddf515eeb9692cb6bb6015: Status 404 returned error can't find the container with id 3b1025752cc7fdae4c6355d02587edb1cc5c96a361ddf515eeb9692cb6bb6015
Jul 27 20:31:43.614191 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.614156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl" event={"ID":"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2","Type":"ContainerStarted","Data":"bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc"}
Jul 27 20:31:43.614368 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.614215    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl" event={"ID":"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2","Type":"ContainerStarted","Data":"6be88922c785c7c456219f044424b64c9cd68e584466504fed178402e229c8d2"}
Jul 27 20:31:43.614368 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.614271    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:43.615469 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.615435    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" event={"ID":"68b6435e-ff87-4882-b54d-01e331c82f4d","Type":"ContainerStarted","Data":"86798ce6430c43a814e2f24b6babe43a57402dd7ef6f1d7884f3d3bf84202bf7"}
Jul 27 20:31:43.616904 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.616883    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j" event={"ID":"5ac4051d-e079-4a5f-a191-c6bc5270aa47","Type":"ContainerStarted","Data":"fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197"}
Jul 27 20:31:43.617006 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.616922    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j" event={"ID":"5ac4051d-e079-4a5f-a191-c6bc5270aa47","Type":"ContainerStarted","Data":"3b1025752cc7fdae4c6355d02587edb1cc5c96a361ddf515eeb9692cb6bb6015"}
Jul 27 20:31:43.617006 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.616952    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:43.633694 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.633650    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl" podStartSLOduration=1.633634794 podStartE2EDuration="1.633634794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:31:43.633018207 +0000 UTC m=+4052.628866213" watchObservedRunningTime="2026-07-27 20:31:43.633634794 +0000 UTC m=+4052.629482769"
Jul 27 20:31:43.648791 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:43.648738    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j" podStartSLOduration=0.648719939 podStartE2EDuration="648.719939ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:31:43.648263356 +0000 UTC m=+4052.644111339" watchObservedRunningTime="2026-07-27 20:31:43.648719939 +0000 UTC m=+4052.644567916"
Jul 27 20:31:44.624461 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:44.624423    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" event={"ID":"68b6435e-ff87-4882-b54d-01e331c82f4d","Type":"ContainerStarted","Data":"9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2"}
Jul 27 20:31:44.624891 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:44.624483    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:44.643293 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:44.643244    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" podStartSLOduration=2.045511159 podStartE2EDuration="2.643230567s" totalImagesPullingTime="597.719408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:31:44.640991836 +0000 UTC m=+4053.636839813" watchObservedRunningTime="2026-07-27 20:31:44.643230567 +0000 UTC m=+4053.639078544"
Jul 27 20:31:46.092755 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:46.092726    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:50.635552 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:50.635509    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:31:54.626252 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:54.626223    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:31:54.626685 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:54.626491    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:31:56.974036 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:56.974001    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:57.266643 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:57.266560    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:31:57.837208 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:31:57.837175    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:02.885567 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:02.885502    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:02.979451 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:02.979417    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:04.887347 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.887313    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"]
Jul 27 20:32:04.890925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.890909    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:04.893380 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.893354    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d68d13-serving-cert\""
Jul 27 20:32:04.902840 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.902819    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"]
Jul 27 20:32:04.921333 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.921299    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:04.953438 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.953398    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:04.953653 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.953447    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8h8l\" (UniqueName: \"kubernetes.io/projected/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-kube-api-access-m8h8l\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:04.953653 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:04.953584    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.054818 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.054785    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.054997 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.054834    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8h8l\" (UniqueName: \"kubernetes.io/projected/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-kube-api-access-m8h8l\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.054997 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.054874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.054997 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:32:05.054948    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-serving-cert: secret "maas-api-e2e-ipp-b-d68d13-serving-cert" not found
Jul 27 20:32:05.055103 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:32:05.055035    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls podName:0c3ff369-203e-45ad-9fb3-adc3bffd3f72 nodeName:}" failed. No retries permitted until 2026-07-27 20:32:05.555013893 +0000 UTC m=+4074.550861856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls") pod "maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" (UID: "0c3ff369-203e-45ad-9fb3-adc3bffd3f72") : secret "maas-api-e2e-ipp-b-d68d13-serving-cert" not found
Jul 27 20:32:05.055495 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.055479    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.066438 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.066409    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8h8l\" (UniqueName: \"kubernetes.io/projected/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-kube-api-access-m8h8l\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.334395 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.334360    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:05.433023 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.432992    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:05.500204 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.500163    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"]
Jul 27 20:32:05.501449 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.501423    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:05.504171 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.504149    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.506679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.506653    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d68d13\""
Jul 27 20:32:05.506799 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.506734    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d68d13-dockercfg-88ch4\""
Jul 27 20:32:05.514461 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.514437    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"]
Jul 27 20:32:05.558886 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.558854    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.561706 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.561683    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.659958 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.659870    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.659958 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.659929    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5njp\" (UniqueName: \"kubernetes.io/projected/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-kube-api-access-k5njp\") pod \"payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.760787 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.760745    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.760968 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.760797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5njp\" (UniqueName: \"kubernetes.io/projected/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-kube-api-access-k5njp\") pod \"payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.761461 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.761441    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.770935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.770910    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5njp\" (UniqueName: \"kubernetes.io/projected/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-kube-api-access-k5njp\") pod \"payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.803702 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.803664    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:05.815994 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.815970    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:05.953948 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.953911    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"]
Jul 27 20:32:05.956604 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:32:05.956573    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3ff369_203e_45ad_9fb3_adc3bffd3f72.slice/crio-aca23a1b6482afe1a93a511fc3eaebcef689c0b79a2b383f95f2f4334ac9ffb8 WatchSource:0}: Error finding container aca23a1b6482afe1a93a511fc3eaebcef689c0b79a2b383f95f2f4334ac9ffb8: Status 404 returned error can't find the container with id aca23a1b6482afe1a93a511fc3eaebcef689c0b79a2b383f95f2f4334ac9ffb8
Jul 27 20:32:05.981831 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:05.981807    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"]
Jul 27 20:32:05.983582 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:32:05.983557    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72d783b_c69e_4a88_be14_0ce7f23ec3a5.slice/crio-7d77f2711bf5650d241b181aeb9693d1e2e5d80041d373538594fa972137846d WatchSource:0}: Error finding container 7d77f2711bf5650d241b181aeb9693d1e2e5d80041d373538594fa972137846d: Status 404 returned error can't find the container with id 7d77f2711bf5650d241b181aeb9693d1e2e5d80041d373538594fa972137846d
Jul 27 20:32:06.748188 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.747997    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" event={"ID":"0c3ff369-203e-45ad-9fb3-adc3bffd3f72","Type":"ContainerStarted","Data":"4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8"}
Jul 27 20:32:06.748188 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.748057    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:06.748188 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.748081    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" event={"ID":"0c3ff369-203e-45ad-9fb3-adc3bffd3f72","Type":"ContainerStarted","Data":"aca23a1b6482afe1a93a511fc3eaebcef689c0b79a2b383f95f2f4334ac9ffb8"}
Jul 27 20:32:06.761858 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.749536    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt" event={"ID":"d72d783b-c69e-4a88-be14-0ce7f23ec3a5","Type":"ContainerStarted","Data":"245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1"}
Jul 27 20:32:06.761858 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.749572    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt" event={"ID":"d72d783b-c69e-4a88-be14-0ce7f23ec3a5","Type":"ContainerStarted","Data":"7d77f2711bf5650d241b181aeb9693d1e2e5d80041d373538594fa972137846d"}
Jul 27 20:32:06.761858 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.749664    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:06.768157 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.768096    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" podStartSLOduration=2.294430723 podStartE2EDuration="2.768074851s" totalImagesPullingTime="473.644128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:32:06.765743886 +0000 UTC m=+4075.761591881" watchObservedRunningTime="2026-07-27 20:32:06.768074851 +0000 UTC m=+4075.763922832"
Jul 27 20:32:06.783171 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:06.783105    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt" podStartSLOduration=1.783085483 podStartE2EDuration="1.783085483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:32:06.781169949 +0000 UTC m=+4075.777017928" watchObservedRunningTime="2026-07-27 20:32:06.783085483 +0000 UTC m=+4075.778933463"
Jul 27 20:32:07.350099 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:07.350063    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:32:12.761067 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:12.761037    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:32:17.757206 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:17.757178    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:32:29.454229 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:29.454198    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dsffp_a5b8f564-3897-4a48-aaa0-4faa317ca28d/payload-processing/0.log"
Jul 27 20:32:44.657616 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:32:44.657585    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:26.567736 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:26.567702    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl_a5ae71d6-4f18-42de-a04a-b0a6fcd125e2/payload-processing/0.log"
Jul 27 20:33:26.694956 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:26.694926    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dsffp_a5b8f564-3897-4a48-aaa0-4faa317ca28d/payload-processing/0.log"
Jul 27 20:33:27.074644 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:27.074614    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:27.400383 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:27.400299    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:27.639596 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:27.639562    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:30.389772 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:30.389738    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:30.493369 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:30.493336    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:32.937735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:32.937696    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"]
Jul 27 20:33:32.942069 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:32.942050    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:32.944611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:32.944591    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-561127-serving-cert\""
Jul 27 20:33:32.953738 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:32.953719    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"]
Jul 27 20:33:32.972809 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:32.972782    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:33.062925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.062880    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztgz\" (UniqueName: \"kubernetes.io/projected/174775db-e600-4740-8cbf-5e269866d147-kube-api-access-pztgz\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.063116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.062950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.063116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.062983    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174775db-e600-4740-8cbf-5e269866d147-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.164437 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.164402    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.164672 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.164447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174775db-e600-4740-8cbf-5e269866d147-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.164672 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:33:33.164566    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-serving-cert: secret "maas-api-e2e-ipp-cleanup-561127-serving-cert" not found
Jul 27 20:33:33.164672 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.164606    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztgz\" (UniqueName: \"kubernetes.io/projected/174775db-e600-4740-8cbf-5e269866d147-kube-api-access-pztgz\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.164672 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:33:33.164625    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls podName:174775db-e600-4740-8cbf-5e269866d147 nodeName:}" failed. No retries permitted until 2026-07-27 20:33:33.6646088 +0000 UTC m=+4162.660456755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" (UID: "174775db-e600-4740-8cbf-5e269866d147") : secret "maas-api-e2e-ipp-cleanup-561127-serving-cert" not found
Jul 27 20:33:33.165121 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.165103    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174775db-e600-4740-8cbf-5e269866d147-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.174709 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.174690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztgz\" (UniqueName: \"kubernetes.io/projected/174775db-e600-4740-8cbf-5e269866d147-kube-api-access-pztgz\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.438958 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.438924    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:33.539682 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.539648    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:33.542940 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.542913    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"]
Jul 27 20:33:33.546672 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.546653    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.549761 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.549738    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-561127-dockercfg-r578k\""
Jul 27 20:33:33.549975 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.549769    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-561127\""
Jul 27 20:33:33.559504 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.559425    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"]
Jul 27 20:33:33.573335 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.573307    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"]
Jul 27 20:33:33.577117 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.577102    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.587402 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.587378    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"]
Jul 27 20:33:33.669226 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.669190    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.669400 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.669258    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.669400 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.669289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw574\" (UniqueName: \"kubernetes.io/projected/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-kube-api-access-mw574\") pod \"payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.669400 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.669373    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.669607 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.669414    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzt6\" (UniqueName: \"kubernetes.io/projected/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-kube-api-access-nlzt6\") pod \"payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.671793 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.671775    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.770272 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.770232    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.770272 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.770277    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw574\" (UniqueName: \"kubernetes.io/projected/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-kube-api-access-mw574\") pod \"payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.770473 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.770321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.770473 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.770348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzt6\" (UniqueName: \"kubernetes.io/projected/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-kube-api-access-nlzt6\") pod \"payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.771019 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.770998    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.771058 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.771030    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.779317 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.779293    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw574\" (UniqueName: \"kubernetes.io/projected/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-kube-api-access-mw574\") pod \"payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:33.779317 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.779309    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzt6\" (UniqueName: \"kubernetes.io/projected/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-kube-api-access-nlzt6\") pod \"payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.853864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.853829    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:33.865909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.865882    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:33.888901 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:33.888877    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:34.019648 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.019621    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"]
Jul 27 20:33:34.022175 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:33:34.021948    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174775db_e600_4740_8cbf_5e269866d147.slice/crio-3c15be62648f26832c970722ab821f0d46be46c6f7cc0f54014ebdd74bdc3be0 WatchSource:0}: Error finding container 3c15be62648f26832c970722ab821f0d46be46c6f7cc0f54014ebdd74bdc3be0: Status 404 returned error can't find the container with id 3c15be62648f26832c970722ab821f0d46be46c6f7cc0f54014ebdd74bdc3be0
Jul 27 20:33:34.078311 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.078288    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"]
Jul 27 20:33:34.080400 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:33:34.080371    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ee4f65_478a_4ba2_9caa_bbee7cd7b612.slice/crio-4a58c887c1668e6af80fe3917a85670baacf1f4c7d8fc53ecccdb9bf9a3255d8 WatchSource:0}: Error finding container 4a58c887c1668e6af80fe3917a85670baacf1f4c7d8fc53ecccdb9bf9a3255d8: Status 404 returned error can't find the container with id 4a58c887c1668e6af80fe3917a85670baacf1f4c7d8fc53ecccdb9bf9a3255d8
Jul 27 20:33:34.200969 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.200934    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" event={"ID":"174775db-e600-4740-8cbf-5e269866d147","Type":"ContainerStarted","Data":"3c15be62648f26832c970722ab821f0d46be46c6f7cc0f54014ebdd74bdc3be0"}
Jul 27 20:33:34.202420 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.202390    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7" event={"ID":"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612","Type":"ContainerStarted","Data":"e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412"}
Jul 27 20:33:34.202559 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.202442    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7" event={"ID":"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612","Type":"ContainerStarted","Data":"4a58c887c1668e6af80fe3917a85670baacf1f4c7d8fc53ecccdb9bf9a3255d8"}
Jul 27 20:33:34.202559 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.202488    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:34.220716 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.220659    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7" podStartSLOduration=1.220639742 podStartE2EDuration="1.220639742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:33:34.219209666 +0000 UTC m=+4163.215057667" watchObservedRunningTime="2026-07-27 20:33:34.220639742 +0000 UTC m=+4163.216487723"
Jul 27 20:33:34.259253 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.259217    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"]
Jul 27 20:33:34.263087 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:33:34.263049    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b47e5c_2048_438b_a1ca_1d6e2eb34f52.slice/crio-60aff0542b3ed6934f021a925814bc27ac7018ea31bd5d64bf3ea71fb997b1f2 WatchSource:0}: Error finding container 60aff0542b3ed6934f021a925814bc27ac7018ea31bd5d64bf3ea71fb997b1f2: Status 404 returned error can't find the container with id 60aff0542b3ed6934f021a925814bc27ac7018ea31bd5d64bf3ea71fb997b1f2
Jul 27 20:33:34.521396 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.521374    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:34.623890 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.623806    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:34.878023 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:34.877952    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:35.209348 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.209251    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x" event={"ID":"97b47e5c-2048-438b-a1ca-1d6e2eb34f52","Type":"ContainerStarted","Data":"95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152"}
Jul 27 20:33:35.209348 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.209307    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x" event={"ID":"97b47e5c-2048-438b-a1ca-1d6e2eb34f52","Type":"ContainerStarted","Data":"60aff0542b3ed6934f021a925814bc27ac7018ea31bd5d64bf3ea71fb997b1f2"}
Jul 27 20:33:35.209878 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.209430    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:35.210815 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.210790    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" event={"ID":"174775db-e600-4740-8cbf-5e269866d147","Type":"ContainerStarted","Data":"5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad"}
Jul 27 20:33:35.210925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.210911    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:35.228059 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.228016    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x" podStartSLOduration=2.228005009 podStartE2EDuration="2.228005009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:33:35.226409138 +0000 UTC m=+4164.222257116" watchObservedRunningTime="2026-07-27 20:33:35.228005009 +0000 UTC m=+4164.223853026"
Jul 27 20:33:35.243040 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:35.242996    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" podStartSLOduration=2.839020332 podStartE2EDuration="3.242982385s" totalImagesPullingTime="403.962053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:33:35.242069879 +0000 UTC m=+4164.237917856" watchObservedRunningTime="2026-07-27 20:33:35.242982385 +0000 UTC m=+4164.238830363"
Jul 27 20:33:41.223228 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:41.223199    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:33:45.213686 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:45.213653    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:33:46.220037 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:46.220011    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:33:50.889328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:50.889287    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"]
Jul 27 20:33:50.894399 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:50.894375    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:50.897040 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:50.897016    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:33:50.900632 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:50.900611    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"]
Jul 27 20:33:51.054078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:51.054036    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/825dbf36-c6e7-4718-a4d1-cdcbabf61346-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428\" (UID: \"825dbf36-c6e7-4718-a4d1-cdcbabf61346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:51.154925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:51.154829    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/825dbf36-c6e7-4718-a4d1-cdcbabf61346-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428\" (UID: \"825dbf36-c6e7-4718-a4d1-cdcbabf61346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:51.162268 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:51.156042    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/825dbf36-c6e7-4718-a4d1-cdcbabf61346-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428\" (UID: \"825dbf36-c6e7-4718-a4d1-cdcbabf61346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:51.207708 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:51.207670    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:51.347566 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:51.347537    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"]
Jul 27 20:33:51.349926 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:33:51.349902    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825dbf36_c6e7_4718_a4d1_cdcbabf61346.slice/crio-9f1a427540843bf2abd9128dda3ab70bd4a6994cd6e1a7db318309cd5cf567bc WatchSource:0}: Error finding container 9f1a427540843bf2abd9128dda3ab70bd4a6994cd6e1a7db318309cd5cf567bc: Status 404 returned error can't find the container with id 9f1a427540843bf2abd9128dda3ab70bd4a6994cd6e1a7db318309cd5cf567bc
Jul 27 20:33:52.308718 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:52.308683    2568 generic.go:356] "Generic (PLEG): container finished" podID="825dbf36-c6e7-4718-a4d1-cdcbabf61346" containerID="66c02fd4da5e9ea7a9f84d7a7c0f653d69cb97488605c9554f09b83b44138497" exitCode=0
Jul 27 20:33:52.309138 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:52.308776    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428" event={"ID":"825dbf36-c6e7-4718-a4d1-cdcbabf61346","Type":"ContainerDied","Data":"66c02fd4da5e9ea7a9f84d7a7c0f653d69cb97488605c9554f09b83b44138497"}
Jul 27 20:33:52.309138 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:52.308845    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428" event={"ID":"825dbf36-c6e7-4718-a4d1-cdcbabf61346","Type":"ContainerStarted","Data":"9f1a427540843bf2abd9128dda3ab70bd4a6994cd6e1a7db318309cd5cf567bc"}
Jul 27 20:33:52.916196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:52.916167    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:33:53.447046 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:53.447023    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:53.578365 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:53.578296    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/825dbf36-c6e7-4718-a4d1-cdcbabf61346-maas-api-service-ca\") pod \"825dbf36-c6e7-4718-a4d1-cdcbabf61346\" (UID: \"825dbf36-c6e7-4718-a4d1-cdcbabf61346\") "
Jul 27 20:33:53.578709 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:53.578690    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825dbf36-c6e7-4718-a4d1-cdcbabf61346-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "825dbf36-c6e7-4718-a4d1-cdcbabf61346" (UID: "825dbf36-c6e7-4718-a4d1-cdcbabf61346"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:33:53.679742 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:53.679716    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/825dbf36-c6e7-4718-a4d1-cdcbabf61346-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:33:54.323934 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:54.323901    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"
Jul 27 20:33:54.323934 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:54.323931    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428" event={"ID":"825dbf36-c6e7-4718-a4d1-cdcbabf61346","Type":"ContainerDied","Data":"9f1a427540843bf2abd9128dda3ab70bd4a6994cd6e1a7db318309cd5cf567bc"}
Jul 27 20:33:54.324148 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:33:54.323979    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f1a427540843bf2abd9128dda3ab70bd4a6994cd6e1a7db318309cd5cf567bc"
Jul 27 20:34:02.816233 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.816197    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"]
Jul 27 20:34:02.816872 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.816482    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" podUID="174775db-e600-4740-8cbf-5e269866d147" containerName="maas-api" containerID="cri-o://5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad" gracePeriod=30
Jul 27 20:34:02.839671 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.839646    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:02.871884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.871854    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"]
Jul 27 20:34:02.872125 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.872078    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7" podUID="a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" containerName="payload-processing" containerID="cri-o://e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412" gracePeriod=30
Jul 27 20:34:02.881332 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.881301    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"]
Jul 27 20:34:02.881672 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.881599    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x" podUID="97b47e5c-2048-438b-a1ca-1d6e2eb34f52" containerName="payload-pre-processing" containerID="cri-o://95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152" gracePeriod=30
Jul 27 20:34:02.923615 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.923583    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:02.983187 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:02.983155    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:03.150383 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.150356    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:34:03.162537 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.162489    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174775db-e600-4740-8cbf-5e269866d147-openshift-service-ca\") pod \"174775db-e600-4740-8cbf-5e269866d147\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") "
Jul 27 20:34:03.162712 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.162557    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pztgz\" (UniqueName: \"kubernetes.io/projected/174775db-e600-4740-8cbf-5e269866d147-kube-api-access-pztgz\") pod \"174775db-e600-4740-8cbf-5e269866d147\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") "
Jul 27 20:34:03.162712 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.162611    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls\") pod \"174775db-e600-4740-8cbf-5e269866d147\" (UID: \"174775db-e600-4740-8cbf-5e269866d147\") "
Jul 27 20:34:03.162943 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.162917    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174775db-e600-4740-8cbf-5e269866d147-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "174775db-e600-4740-8cbf-5e269866d147" (UID: "174775db-e600-4740-8cbf-5e269866d147"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:03.163074 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.163059    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/174775db-e600-4740-8cbf-5e269866d147-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.165894 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.165866    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "174775db-e600-4740-8cbf-5e269866d147" (UID: "174775db-e600-4740-8cbf-5e269866d147"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:34:03.166981 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.166957    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174775db-e600-4740-8cbf-5e269866d147-kube-api-access-pztgz" (OuterVolumeSpecName: "kube-api-access-pztgz") pod "174775db-e600-4740-8cbf-5e269866d147" (UID: "174775db-e600-4740-8cbf-5e269866d147"). InnerVolumeSpecName "kube-api-access-pztgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:34:03.179498 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.179472    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:34:03.192762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.192738    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:34:03.264139 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264106    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlzt6\" (UniqueName: \"kubernetes.io/projected/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-kube-api-access-nlzt6\") pod \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") "
Jul 27 20:34:03.264310 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264144    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-plugins-config-volume\") pod \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\" (UID: \"97b47e5c-2048-438b-a1ca-1d6e2eb34f52\") "
Jul 27 20:34:03.264310 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264178    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-plugins-config-volume\") pod \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") "
Jul 27 20:34:03.264310 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264286    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw574\" (UniqueName: \"kubernetes.io/projected/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-kube-api-access-mw574\") pod \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\" (UID: \"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612\") "
Jul 27 20:34:03.264605 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264580    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" (UID: "a2ee4f65-478a-4ba2-9caa-bbee7cd7b612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:03.264723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264584    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97b47e5c-2048-438b-a1ca-1d6e2eb34f52" (UID: "97b47e5c-2048-438b-a1ca-1d6e2eb34f52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:03.264723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264691    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/174775db-e600-4740-8cbf-5e269866d147-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.264723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264706    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.264723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.264717    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztgz\" (UniqueName: \"kubernetes.io/projected/174775db-e600-4740-8cbf-5e269866d147-kube-api-access-pztgz\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.266335 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.266316    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-kube-api-access-nlzt6" (OuterVolumeSpecName: "kube-api-access-nlzt6") pod "97b47e5c-2048-438b-a1ca-1d6e2eb34f52" (UID: "97b47e5c-2048-438b-a1ca-1d6e2eb34f52"). InnerVolumeSpecName "kube-api-access-nlzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:34:03.266436 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.266416    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-kube-api-access-mw574" (OuterVolumeSpecName: "kube-api-access-mw574") pod "a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" (UID: "a2ee4f65-478a-4ba2-9caa-bbee7cd7b612"). InnerVolumeSpecName "kube-api-access-mw574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:34:03.366038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.365998    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw574\" (UniqueName: \"kubernetes.io/projected/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612-kube-api-access-mw574\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.366038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.366035    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzt6\" (UniqueName: \"kubernetes.io/projected/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-kube-api-access-nlzt6\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.366238 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.366050    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b47e5c-2048-438b-a1ca-1d6e2eb34f52-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:03.371743 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.371667    2568 generic.go:356] "Generic (PLEG): container finished" podID="97b47e5c-2048-438b-a1ca-1d6e2eb34f52" containerID="95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152" exitCode=0
Jul 27 20:34:03.371743 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.371721    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"
Jul 27 20:34:03.371959 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.371753    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x" event={"ID":"97b47e5c-2048-438b-a1ca-1d6e2eb34f52","Type":"ContainerDied","Data":"95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152"}
Jul 27 20:34:03.371959 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.371849    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x" event={"ID":"97b47e5c-2048-438b-a1ca-1d6e2eb34f52","Type":"ContainerDied","Data":"60aff0542b3ed6934f021a925814bc27ac7018ea31bd5d64bf3ea71fb997b1f2"}
Jul 27 20:34:03.371959 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.371879    2568 scope.go:117] "RemoveContainer" containerID="95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152"
Jul 27 20:34:03.373304 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.373277    2568 generic.go:356] "Generic (PLEG): container finished" podID="174775db-e600-4740-8cbf-5e269866d147" containerID="5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad" exitCode=0
Jul 27 20:34:03.373406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.373318    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"
Jul 27 20:34:03.373406 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.373366    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" event={"ID":"174775db-e600-4740-8cbf-5e269866d147","Type":"ContainerDied","Data":"5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad"}
Jul 27 20:34:03.373509 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.373425    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5" event={"ID":"174775db-e600-4740-8cbf-5e269866d147","Type":"ContainerDied","Data":"3c15be62648f26832c970722ab821f0d46be46c6f7cc0f54014ebdd74bdc3be0"}
Jul 27 20:34:03.374923 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.374900    2568 generic.go:356] "Generic (PLEG): container finished" podID="a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" containerID="e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412" exitCode=0
Jul 27 20:34:03.375039 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.374942    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"
Jul 27 20:34:03.375039 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.374971    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7" event={"ID":"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612","Type":"ContainerDied","Data":"e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412"}
Jul 27 20:34:03.375039 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.375032    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7" event={"ID":"a2ee4f65-478a-4ba2-9caa-bbee7cd7b612","Type":"ContainerDied","Data":"4a58c887c1668e6af80fe3917a85670baacf1f4c7d8fc53ecccdb9bf9a3255d8"}
Jul 27 20:34:03.383056 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.383033    2568 scope.go:117] "RemoveContainer" containerID="95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152"
Jul 27 20:34:03.383321 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:34:03.383303    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152\": container with ID starting with 95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152 not found: ID does not exist" containerID="95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152"
Jul 27 20:34:03.383401 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.383330    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152"} err="failed to get container status \"95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152\": rpc error: code = NotFound desc = could not find container \"95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152\": container with ID starting with 95d3f15fb339ffab45a275d2561161866a401d1aec1f934b916ef80e106d1152 not found: ID does not exist"
Jul 27 20:34:03.383401 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.383347    2568 scope.go:117] "RemoveContainer" containerID="5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad"
Jul 27 20:34:03.392375 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.392356    2568 scope.go:117] "RemoveContainer" containerID="5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad"
Jul 27 20:34:03.392649 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:34:03.392633    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad\": container with ID starting with 5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad not found: ID does not exist" containerID="5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad"
Jul 27 20:34:03.392696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.392655    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad"} err="failed to get container status \"5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad\": rpc error: code = NotFound desc = could not find container \"5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad\": container with ID starting with 5f782a1f3251d18d20cbec4b69b618da2de09092e68fc6f33f39e770994ea4ad not found: ID does not exist"
Jul 27 20:34:03.392696 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.392672    2568 scope.go:117] "RemoveContainer" containerID="e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412"
Jul 27 20:34:03.398601 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.398580    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"]
Jul 27 20:34:03.401791 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.401742    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-561127-5d688ff7d8-p4n8x"]
Jul 27 20:34:03.401851 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.401833    2568 scope.go:117] "RemoveContainer" containerID="e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412"
Jul 27 20:34:03.402106 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:34:03.402089    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412\": container with ID starting with e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412 not found: ID does not exist" containerID="e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412"
Jul 27 20:34:03.402165 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.402111    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412"} err="failed to get container status \"e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412\": rpc error: code = NotFound desc = could not find container \"e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412\": container with ID starting with e1e3cd68dc17a156e175fc2d560bfb7ebb74915bf360e2b487285f19e707a412 not found: ID does not exist"
Jul 27 20:34:03.413213 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.413192    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"]
Jul 27 20:34:03.418294 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.418271    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-561127-67fd99d8cb-pnvx7"]
Jul 27 20:34:03.430124 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.430102    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"]
Jul 27 20:34:03.438603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.438584    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-561127-5f8bf75f98-56bm5"]
Jul 27 20:34:03.568704 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.568675    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174775db-e600-4740-8cbf-5e269866d147" path="/var/lib/kubelet/pods/174775db-e600-4740-8cbf-5e269866d147/volumes"
Jul 27 20:34:03.569015 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.569002    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b47e5c-2048-438b-a1ca-1d6e2eb34f52" path="/var/lib/kubelet/pods/97b47e5c-2048-438b-a1ca-1d6e2eb34f52/volumes"
Jul 27 20:34:03.569304 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:03.569292    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" path="/var/lib/kubelet/pods/a2ee4f65-478a-4ba2-9caa-bbee7cd7b612/volumes"
Jul 27 20:34:05.966163 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:05.966131    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"]
Jul 27 20:34:05.968279 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:05.968259    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-561127-dz428"]
Jul 27 20:34:05.970487 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:05.970458    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:06.328461 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:06.328427    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:07.568383 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:07.568349    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825dbf36-c6e7-4718-a4d1-cdcbabf61346" path="/var/lib/kubelet/pods/825dbf36-c6e7-4718-a4d1-cdcbabf61346/volumes"
Jul 27 20:34:11.782472 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:11.782448    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:34:11.809985 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:11.809959    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:34:13.194609 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:13.194562    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:13.295082 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:13.295035    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:13.445847 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:13.445760    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:13.667664 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:13.667624    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:16.947249 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947218    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"]
Jul 27 20:34:16.947670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947658    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825dbf36-c6e7-4718-a4d1-cdcbabf61346" containerName="revoke-keys"
Jul 27 20:34:16.947670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947672    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="825dbf36-c6e7-4718-a4d1-cdcbabf61346" containerName="revoke-keys"
Jul 27 20:34:16.947745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947683    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174775db-e600-4740-8cbf-5e269866d147" containerName="maas-api"
Jul 27 20:34:16.947745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947688    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="174775db-e600-4740-8cbf-5e269866d147" containerName="maas-api"
Jul 27 20:34:16.947745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947701    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b47e5c-2048-438b-a1ca-1d6e2eb34f52" containerName="payload-pre-processing"
Jul 27 20:34:16.947745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947707    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b47e5c-2048-438b-a1ca-1d6e2eb34f52" containerName="payload-pre-processing"
Jul 27 20:34:16.947745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947716    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" containerName="payload-processing"
Jul 27 20:34:16.947745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947721    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" containerName="payload-processing"
Jul 27 20:34:16.947932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947791    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b47e5c-2048-438b-a1ca-1d6e2eb34f52" containerName="payload-pre-processing"
Jul 27 20:34:16.947932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947799    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2ee4f65-478a-4ba2-9caa-bbee7cd7b612" containerName="payload-processing"
Jul 27 20:34:16.947932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947808    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="825dbf36-c6e7-4718-a4d1-cdcbabf61346" containerName="revoke-keys"
Jul 27 20:34:16.947932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.947814    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="174775db-e600-4740-8cbf-5e269866d147" containerName="maas-api"
Jul 27 20:34:16.952365 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.952350    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:16.955608 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.955587    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:34:16.959253 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.959220    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"]
Jul 27 20:34:16.997029 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:16.996994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9581fc05-ebbd-4409-926a-59afe1cf4fa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8\" (UID: \"9581fc05-ebbd-4409-926a-59afe1cf4fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:17.097971 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:17.097936    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9581fc05-ebbd-4409-926a-59afe1cf4fa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8\" (UID: \"9581fc05-ebbd-4409-926a-59afe1cf4fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:17.098640 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:17.098620    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9581fc05-ebbd-4409-926a-59afe1cf4fa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8\" (UID: \"9581fc05-ebbd-4409-926a-59afe1cf4fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:17.265330 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:17.265251    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:17.407929 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:17.407898    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"]
Jul 27 20:34:17.409952 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:34:17.409917    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9581fc05_ebbd_4409_926a_59afe1cf4fa1.slice/crio-2976fb76f242e13fc253fd0474a8036ec9d5863958426e365942d551e00e1b53 WatchSource:0}: Error finding container 2976fb76f242e13fc253fd0474a8036ec9d5863958426e365942d551e00e1b53: Status 404 returned error can't find the container with id 2976fb76f242e13fc253fd0474a8036ec9d5863958426e365942d551e00e1b53
Jul 27 20:34:17.452806 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:17.452777    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8" event={"ID":"9581fc05-ebbd-4409-926a-59afe1cf4fa1","Type":"ContainerStarted","Data":"2976fb76f242e13fc253fd0474a8036ec9d5863958426e365942d551e00e1b53"}
Jul 27 20:34:18.460955 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:18.460864    2568 generic.go:356] "Generic (PLEG): container finished" podID="9581fc05-ebbd-4409-926a-59afe1cf4fa1" containerID="584a8f5cee320c225e5d869a623e43049d3855c10f07493c5b592581df1b6507" exitCode=0
Jul 27 20:34:18.460955 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:18.460896    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8" event={"ID":"9581fc05-ebbd-4409-926a-59afe1cf4fa1","Type":"ContainerDied","Data":"584a8f5cee320c225e5d869a623e43049d3855c10f07493c5b592581df1b6507"}
Jul 27 20:34:19.607253 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:19.607231    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:19.724173 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:19.724053    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9581fc05-ebbd-4409-926a-59afe1cf4fa1-maas-api-service-ca\") pod \"9581fc05-ebbd-4409-926a-59afe1cf4fa1\" (UID: \"9581fc05-ebbd-4409-926a-59afe1cf4fa1\") "
Jul 27 20:34:19.724453 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:19.724432    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9581fc05-ebbd-4409-926a-59afe1cf4fa1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9581fc05-ebbd-4409-926a-59afe1cf4fa1" (UID: "9581fc05-ebbd-4409-926a-59afe1cf4fa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:19.825755 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:19.825722    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9581fc05-ebbd-4409-926a-59afe1cf4fa1-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:20.472618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:20.472578    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8" event={"ID":"9581fc05-ebbd-4409-926a-59afe1cf4fa1","Type":"ContainerDied","Data":"2976fb76f242e13fc253fd0474a8036ec9d5863958426e365942d551e00e1b53"}
Jul 27 20:34:20.472618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:20.472596    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"
Jul 27 20:34:20.472831 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:20.472629    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2976fb76f242e13fc253fd0474a8036ec9d5863958426e365942d551e00e1b53"
Jul 27 20:34:21.415103 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:21.415068    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:24.845989 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:24.845955    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:37.653317 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.653285    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"]
Jul 27 20:34:37.653768 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.653507    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" podUID="68b6435e-ff87-4882-b54d-01e331c82f4d" containerName="maas-api" containerID="cri-o://9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2" gracePeriod=30
Jul 27 20:34:37.664106 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.664082    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:37.731688 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.731656    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"]
Jul 27 20:34:37.731923 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.731878    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl" podUID="a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" containerName="payload-processing" containerID="cri-o://bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc" gracePeriod=30
Jul 27 20:34:37.739101 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.739074    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"]
Jul 27 20:34:37.739267 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.739249    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j" podUID="5ac4051d-e079-4a5f-a191-c6bc5270aa47" containerName="payload-pre-processing" containerID="cri-o://fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197" gracePeriod=30
Jul 27 20:34:37.832112 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.832077    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:37.931444 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:37.931414    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:38.017519 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.017494    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:34:38.041931 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.041886    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:34:38.052847 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.052826    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:34:38.204752 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204668    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b6435e-ff87-4882-b54d-01e331c82f4d-openshift-service-ca\") pod \"68b6435e-ff87-4882-b54d-01e331c82f4d\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") "
Jul 27 20:34:38.204915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204773    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v4kj\" (UniqueName: \"kubernetes.io/projected/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-kube-api-access-8v4kj\") pod \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") "
Jul 27 20:34:38.204915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204810    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac4051d-e079-4a5f-a191-c6bc5270aa47-plugins-config-volume\") pod \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") "
Jul 27 20:34:38.204915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204846    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4ppf\" (UniqueName: \"kubernetes.io/projected/5ac4051d-e079-4a5f-a191-c6bc5270aa47-kube-api-access-r4ppf\") pod \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\" (UID: \"5ac4051d-e079-4a5f-a191-c6bc5270aa47\") "
Jul 27 20:34:38.204915 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204897    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-plugins-config-volume\") pod \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\" (UID: \"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2\") "
Jul 27 20:34:38.205147 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204942    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls\") pod \"68b6435e-ff87-4882-b54d-01e331c82f4d\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") "
Jul 27 20:34:38.205147 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.204969    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvt28\" (UniqueName: \"kubernetes.io/projected/68b6435e-ff87-4882-b54d-01e331c82f4d-kube-api-access-vvt28\") pod \"68b6435e-ff87-4882-b54d-01e331c82f4d\" (UID: \"68b6435e-ff87-4882-b54d-01e331c82f4d\") "
Jul 27 20:34:38.205283 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.205184    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b6435e-ff87-4882-b54d-01e331c82f4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68b6435e-ff87-4882-b54d-01e331c82f4d" (UID: "68b6435e-ff87-4882-b54d-01e331c82f4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:38.205283 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.205212    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac4051d-e079-4a5f-a191-c6bc5270aa47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ac4051d-e079-4a5f-a191-c6bc5270aa47" (UID: "5ac4051d-e079-4a5f-a191-c6bc5270aa47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:38.205362 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.205274    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" (UID: "a5ae71d6-4f18-42de-a04a-b0a6fcd125e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:34:38.205362 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.205341    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b6435e-ff87-4882-b54d-01e331c82f4d-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.205439 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.205362    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac4051d-e079-4a5f-a191-c6bc5270aa47-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.207304 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.207268    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68b6435e-ff87-4882-b54d-01e331c82f4d" (UID: "68b6435e-ff87-4882-b54d-01e331c82f4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:34:38.207304 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.207278    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-kube-api-access-8v4kj" (OuterVolumeSpecName: "kube-api-access-8v4kj") pod "a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" (UID: "a5ae71d6-4f18-42de-a04a-b0a6fcd125e2"). InnerVolumeSpecName "kube-api-access-8v4kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:34:38.207445 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.207315    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b6435e-ff87-4882-b54d-01e331c82f4d-kube-api-access-vvt28" (OuterVolumeSpecName: "kube-api-access-vvt28") pod "68b6435e-ff87-4882-b54d-01e331c82f4d" (UID: "68b6435e-ff87-4882-b54d-01e331c82f4d"). InnerVolumeSpecName "kube-api-access-vvt28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:34:38.207445 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.207342    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac4051d-e079-4a5f-a191-c6bc5270aa47-kube-api-access-r4ppf" (OuterVolumeSpecName: "kube-api-access-r4ppf") pod "5ac4051d-e079-4a5f-a191-c6bc5270aa47" (UID: "5ac4051d-e079-4a5f-a191-c6bc5270aa47"). InnerVolumeSpecName "kube-api-access-r4ppf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:34:38.306459 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.306430    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v4kj\" (UniqueName: \"kubernetes.io/projected/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-kube-api-access-8v4kj\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.306459 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.306457    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ppf\" (UniqueName: \"kubernetes.io/projected/5ac4051d-e079-4a5f-a191-c6bc5270aa47-kube-api-access-r4ppf\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.306459 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.306468    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.306714 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.306478    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b6435e-ff87-4882-b54d-01e331c82f4d-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.306714 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.306486    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvt28\" (UniqueName: \"kubernetes.io/projected/68b6435e-ff87-4882-b54d-01e331c82f4d-kube-api-access-vvt28\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:34:38.568158 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.568128    2568 generic.go:356] "Generic (PLEG): container finished" podID="a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" containerID="bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc" exitCode=0
Jul 27 20:34:38.568344 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.568179    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"
Jul 27 20:34:38.568344 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.568206    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl" event={"ID":"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2","Type":"ContainerDied","Data":"bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc"}
Jul 27 20:34:38.568344 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.568256    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl" event={"ID":"a5ae71d6-4f18-42de-a04a-b0a6fcd125e2","Type":"ContainerDied","Data":"6be88922c785c7c456219f044424b64c9cd68e584466504fed178402e229c8d2"}
Jul 27 20:34:38.568344 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.568281    2568 scope.go:117] "RemoveContainer" containerID="bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc"
Jul 27 20:34:38.569469 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.569446    2568 generic.go:356] "Generic (PLEG): container finished" podID="68b6435e-ff87-4882-b54d-01e331c82f4d" containerID="9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2" exitCode=0
Jul 27 20:34:38.569564 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.569501    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"
Jul 27 20:34:38.569564 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.569545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" event={"ID":"68b6435e-ff87-4882-b54d-01e331c82f4d","Type":"ContainerDied","Data":"9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2"}
Jul 27 20:34:38.569669 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.569587    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc" event={"ID":"68b6435e-ff87-4882-b54d-01e331c82f4d","Type":"ContainerDied","Data":"86798ce6430c43a814e2f24b6babe43a57402dd7ef6f1d7884f3d3bf84202bf7"}
Jul 27 20:34:38.571018 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.570993    2568 generic.go:356] "Generic (PLEG): container finished" podID="5ac4051d-e079-4a5f-a191-c6bc5270aa47" containerID="fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197" exitCode=0
Jul 27 20:34:38.571147 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.571028    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"
Jul 27 20:34:38.571147 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.571047    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j" event={"ID":"5ac4051d-e079-4a5f-a191-c6bc5270aa47","Type":"ContainerDied","Data":"fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197"}
Jul 27 20:34:38.571147 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.571088    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j" event={"ID":"5ac4051d-e079-4a5f-a191-c6bc5270aa47","Type":"ContainerDied","Data":"3b1025752cc7fdae4c6355d02587edb1cc5c96a361ddf515eeb9692cb6bb6015"}
Jul 27 20:34:38.578972 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.578950    2568 scope.go:117] "RemoveContainer" containerID="bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc"
Jul 27 20:34:38.579209 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:34:38.579191    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc\": container with ID starting with bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc not found: ID does not exist" containerID="bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc"
Jul 27 20:34:38.579289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.579216    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc"} err="failed to get container status \"bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc\": rpc error: code = NotFound desc = could not find container \"bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc\": container with ID starting with bf3eb61c04bdfe60c303df7a8387c7302ddddc5da08b7c3e292b5725b6dfbbdc not found: ID does not exist"
Jul 27 20:34:38.579289 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.579233    2568 scope.go:117] "RemoveContainer" containerID="9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2"
Jul 27 20:34:38.587555 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.587511    2568 scope.go:117] "RemoveContainer" containerID="9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2"
Jul 27 20:34:38.587826 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:34:38.587804    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2\": container with ID starting with 9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2 not found: ID does not exist" containerID="9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2"
Jul 27 20:34:38.587913 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.587833    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2"} err="failed to get container status \"9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2\": rpc error: code = NotFound desc = could not find container \"9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2\": container with ID starting with 9d768e91a679f44dd9b8dc9be22c4ade83f6baf86e9c0f5cfbafde563c18adb2 not found: ID does not exist"
Jul 27 20:34:38.587913 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.587858    2568 scope.go:117] "RemoveContainer" containerID="fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197"
Jul 27 20:34:38.595920 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.595897    2568 scope.go:117] "RemoveContainer" containerID="fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197"
Jul 27 20:34:38.596192 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:34:38.596173    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197\": container with ID starting with fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197 not found: ID does not exist" containerID="fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197"
Jul 27 20:34:38.596281 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.596197    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197"} err="failed to get container status \"fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197\": rpc error: code = NotFound desc = could not find container \"fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197\": container with ID starting with fea280bcf956d1f643688f508c543dc6c091dc381b469e2bd45d54607ffb8197 not found: ID does not exist"
Jul 27 20:34:38.597113 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.597094    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"]
Jul 27 20:34:38.601220 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.601202    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac78b-886b8866-b7dfc"]
Jul 27 20:34:38.614909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.614887    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"]
Jul 27 20:34:38.618480 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.618459    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac78b-698f58c784-brzsl"]
Jul 27 20:34:38.631022 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.630999    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"]
Jul 27 20:34:38.635401 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:38.635382    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ac78b-645f64f8fb-hfb9j"]
Jul 27 20:34:39.568577 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:39.568549    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac4051d-e079-4a5f-a191-c6bc5270aa47" path="/var/lib/kubelet/pods/5ac4051d-e079-4a5f-a191-c6bc5270aa47/volumes"
Jul 27 20:34:39.568943 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:39.568917    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b6435e-ff87-4882-b54d-01e331c82f4d" path="/var/lib/kubelet/pods/68b6435e-ff87-4882-b54d-01e331c82f4d/volumes"
Jul 27 20:34:39.569216 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:39.569204    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" path="/var/lib/kubelet/pods/a5ae71d6-4f18-42de-a04a-b0a6fcd125e2/volumes"
Jul 27 20:34:42.025383 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:42.025343    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:42.026386 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:42.026364    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"]
Jul 27 20:34:42.028353 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:42.028333    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ac78b-49qn8"]
Jul 27 20:34:42.389001 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:42.388923    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:34:43.567656 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:43.567623    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9581fc05-ebbd-4409-926a-59afe1cf4fa1" path="/var/lib/kubelet/pods/9581fc05-ebbd-4409-926a-59afe1cf4fa1/volumes"
Jul 27 20:34:47.572011 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:34:47.571980    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:25.342744 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:25.342704    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:26.506269 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:26.506223    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:33.774862 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:33.774816    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:40.552260 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:40.552226    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:40.652332 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:40.652298    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:40.821128 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:40.821040    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:41.040187 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.040146    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:41.154829 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.154744    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"]
Jul 27 20:35:41.155166 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155154    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b6435e-ff87-4882-b54d-01e331c82f4d" containerName="maas-api"
Jul 27 20:35:41.155211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155168    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b6435e-ff87-4882-b54d-01e331c82f4d" containerName="maas-api"
Jul 27 20:35:41.155211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155187    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac4051d-e079-4a5f-a191-c6bc5270aa47" containerName="payload-pre-processing"
Jul 27 20:35:41.155211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155192    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac4051d-e079-4a5f-a191-c6bc5270aa47" containerName="payload-pre-processing"
Jul 27 20:35:41.155211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155200    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9581fc05-ebbd-4409-926a-59afe1cf4fa1" containerName="revoke-keys"
Jul 27 20:35:41.155211 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155205    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9581fc05-ebbd-4409-926a-59afe1cf4fa1" containerName="revoke-keys"
Jul 27 20:35:41.155373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155215    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" containerName="payload-processing"
Jul 27 20:35:41.155373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155221    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" containerName="payload-processing"
Jul 27 20:35:41.155373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155290    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b6435e-ff87-4882-b54d-01e331c82f4d" containerName="maas-api"
Jul 27 20:35:41.155373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155301    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ae71d6-4f18-42de-a04a-b0a6fcd125e2" containerName="payload-processing"
Jul 27 20:35:41.155373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155307    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9581fc05-ebbd-4409-926a-59afe1cf4fa1" containerName="revoke-keys"
Jul 27 20:35:41.155373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.155315    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac4051d-e079-4a5f-a191-c6bc5270aa47" containerName="payload-pre-processing"
Jul 27 20:35:41.158366 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.158351    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:41.161024 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.161008    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:35:41.166622 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.166599    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"]
Jul 27 20:35:41.293865 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.293834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p\" (UID: \"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:41.395228 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.395196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p\" (UID: \"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:41.395864 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.395838    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p\" (UID: \"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:41.469722 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.469622    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:41.611536 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.611499    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"]
Jul 27 20:35:41.613934 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:35:41.613901    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd41c27_d3ec_4e43_ae53_3f2eac44dc21.slice/crio-9bfd3e3a134868de13cf70f689dab67330a4c0c80b6fe7e2778880f4834f5503 WatchSource:0}: Error finding container 9bfd3e3a134868de13cf70f689dab67330a4c0c80b6fe7e2778880f4834f5503: Status 404 returned error can't find the container with id 9bfd3e3a134868de13cf70f689dab67330a4c0c80b6fe7e2778880f4834f5503
Jul 27 20:35:41.901781 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.901746    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p" event={"ID":"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21","Type":"ContainerStarted","Data":"3f37286d0f099601fc4e7090886cdb5988f360536888b90c87769c0743f1dfaf"}
Jul 27 20:35:41.901936 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.901805    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p" event={"ID":"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21","Type":"ContainerStarted","Data":"9bfd3e3a134868de13cf70f689dab67330a4c0c80b6fe7e2778880f4834f5503"}
Jul 27 20:35:41.918493 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:41.918448    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p" podStartSLOduration=0.918432935 podStartE2EDuration="918.432935ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:35:41.917395841 +0000 UTC m=+4290.913243819" watchObservedRunningTime="2026-07-27 20:35:41.918432935 +0000 UTC m=+4290.914280957"
Jul 27 20:35:42.908877 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:42.908837    2568 generic.go:356] "Generic (PLEG): container finished" podID="3fd41c27-d3ec-4e43-ae53-3f2eac44dc21" containerID="3f37286d0f099601fc4e7090886cdb5988f360536888b90c87769c0743f1dfaf" exitCode=0
Jul 27 20:35:42.909331 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:42.908903    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p" event={"ID":"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21","Type":"ContainerDied","Data":"3f37286d0f099601fc4e7090886cdb5988f360536888b90c87769c0743f1dfaf"}
Jul 27 20:35:44.051036 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.051013    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:44.126789 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.126756    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21-maas-api-service-ca\") pod \"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21\" (UID: \"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21\") "
Jul 27 20:35:44.127149 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.127124    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fd41c27-d3ec-4e43-ae53-3f2eac44dc21" (UID: "3fd41c27-d3ec-4e43-ae53-3f2eac44dc21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:35:44.228434 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.228349    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:35:44.920054 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.920020    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"
Jul 27 20:35:44.920228 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.920019    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p" event={"ID":"3fd41c27-d3ec-4e43-ae53-3f2eac44dc21","Type":"ContainerDied","Data":"9bfd3e3a134868de13cf70f689dab67330a4c0c80b6fe7e2778880f4834f5503"}
Jul 27 20:35:44.920228 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:44.920153    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bfd3e3a134868de13cf70f689dab67330a4c0c80b6fe7e2778880f4834f5503"
Jul 27 20:35:52.282979 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.282944    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"]
Jul 27 20:35:52.283436 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.283140    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" podUID="0c3ff369-203e-45ad-9fb3-adc3bffd3f72" containerName="maas-api" containerID="cri-o://4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8" gracePeriod=30
Jul 27 20:35:52.303786 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.303725    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:52.385124 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.385091    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"]
Jul 27 20:35:52.385376 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.385350    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt" podUID="d72d783b-c69e-4a88-be14-0ce7f23ec3a5" containerName="payload-pre-processing" containerID="cri-o://245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1" gracePeriod=30
Jul 27 20:35:52.417516 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.417491    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:52.510733 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.510703    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:52.553961 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.553938    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:35:52.660645 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.660622    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:35:52.705492 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.705461    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8h8l\" (UniqueName: \"kubernetes.io/projected/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-kube-api-access-m8h8l\") pod \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") "
Jul 27 20:35:52.705707 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.705507    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls\") pod \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") "
Jul 27 20:35:52.705707 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.705568    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-openshift-service-ca\") pod \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\" (UID: \"0c3ff369-203e-45ad-9fb3-adc3bffd3f72\") "
Jul 27 20:35:52.706000 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.705976    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c3ff369-203e-45ad-9fb3-adc3bffd3f72" (UID: "0c3ff369-203e-45ad-9fb3-adc3bffd3f72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:35:52.707745 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.707723    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c3ff369-203e-45ad-9fb3-adc3bffd3f72" (UID: "0c3ff369-203e-45ad-9fb3-adc3bffd3f72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:35:52.707832 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.707785    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-kube-api-access-m8h8l" (OuterVolumeSpecName: "kube-api-access-m8h8l") pod "0c3ff369-203e-45ad-9fb3-adc3bffd3f72" (UID: "0c3ff369-203e-45ad-9fb3-adc3bffd3f72"). InnerVolumeSpecName "kube-api-access-m8h8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:35:52.806740 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.806645    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-plugins-config-volume\") pod \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") "
Jul 27 20:35:52.806740 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.806717    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5njp\" (UniqueName: \"kubernetes.io/projected/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-kube-api-access-k5njp\") pod \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\" (UID: \"d72d783b-c69e-4a88-be14-0ce7f23ec3a5\") "
Jul 27 20:35:52.807024 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.807001    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d72d783b-c69e-4a88-be14-0ce7f23ec3a5" (UID: "d72d783b-c69e-4a88-be14-0ce7f23ec3a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:35:52.807087 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.807068    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8h8l\" (UniqueName: \"kubernetes.io/projected/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-kube-api-access-m8h8l\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:35:52.807140 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.807088    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:35:52.807140 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.807102    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c3ff369-203e-45ad-9fb3-adc3bffd3f72-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:35:52.808837 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.808817    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-kube-api-access-k5njp" (OuterVolumeSpecName: "kube-api-access-k5njp") pod "d72d783b-c69e-4a88-be14-0ce7f23ec3a5" (UID: "d72d783b-c69e-4a88-be14-0ce7f23ec3a5"). InnerVolumeSpecName "kube-api-access-k5njp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:35:52.907588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.907557    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:35:52.907588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.907584    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5njp\" (UniqueName: \"kubernetes.io/projected/d72d783b-c69e-4a88-be14-0ce7f23ec3a5-kube-api-access-k5njp\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:35:52.964302 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.964263    2568 generic.go:356] "Generic (PLEG): container finished" podID="d72d783b-c69e-4a88-be14-0ce7f23ec3a5" containerID="245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1" exitCode=0
Jul 27 20:35:52.964500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.964306    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"
Jul 27 20:35:52.964500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.964352    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt" event={"ID":"d72d783b-c69e-4a88-be14-0ce7f23ec3a5","Type":"ContainerDied","Data":"245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1"}
Jul 27 20:35:52.964500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.964409    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt" event={"ID":"d72d783b-c69e-4a88-be14-0ce7f23ec3a5","Type":"ContainerDied","Data":"7d77f2711bf5650d241b181aeb9693d1e2e5d80041d373538594fa972137846d"}
Jul 27 20:35:52.964500 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.964445    2568 scope.go:117] "RemoveContainer" containerID="245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1"
Jul 27 20:35:52.965909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.965681    2568 generic.go:356] "Generic (PLEG): container finished" podID="0c3ff369-203e-45ad-9fb3-adc3bffd3f72" containerID="4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8" exitCode=0
Jul 27 20:35:52.965909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.965715    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" event={"ID":"0c3ff369-203e-45ad-9fb3-adc3bffd3f72","Type":"ContainerDied","Data":"4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8"}
Jul 27 20:35:52.965909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.965787    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx" event={"ID":"0c3ff369-203e-45ad-9fb3-adc3bffd3f72","Type":"ContainerDied","Data":"aca23a1b6482afe1a93a511fc3eaebcef689c0b79a2b383f95f2f4334ac9ffb8"}
Jul 27 20:35:52.965909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.965728    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"
Jul 27 20:35:52.974448 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.974426    2568 scope.go:117] "RemoveContainer" containerID="245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1"
Jul 27 20:35:52.974779 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:35:52.974761    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1\": container with ID starting with 245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1 not found: ID does not exist" containerID="245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1"
Jul 27 20:35:52.974893 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.974789    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1"} err="failed to get container status \"245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1\": rpc error: code = NotFound desc = could not find container \"245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1\": container with ID starting with 245de6a5e2aa8df509539292752752939903bbe25e800288d968367d1cb33ae1 not found: ID does not exist"
Jul 27 20:35:52.974893 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.974806    2568 scope.go:117] "RemoveContainer" containerID="4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8"
Jul 27 20:35:52.984801 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.984785    2568 scope.go:117] "RemoveContainer" containerID="4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8"
Jul 27 20:35:52.985064 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:35:52.985044    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8\": container with ID starting with 4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8 not found: ID does not exist" containerID="4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8"
Jul 27 20:35:52.985115 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.985073    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8"} err="failed to get container status \"4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8\": rpc error: code = NotFound desc = could not find container \"4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8\": container with ID starting with 4e022eb5f1a587a580ecc0132a647d8323ea7b8f200a2798c324f9007591f2a8 not found: ID does not exist"
Jul 27 20:35:52.991824 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.991796    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"]
Jul 27 20:35:52.993859 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:52.993841    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d68d13-76bbf5d584-5ptdt"]
Jul 27 20:35:53.006312 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:53.006291    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"]
Jul 27 20:35:53.010572 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:53.010552    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d68d13-5c4849b55b-s9hsx"]
Jul 27 20:35:53.567752 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:53.567677    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3ff369-203e-45ad-9fb3-adc3bffd3f72" path="/var/lib/kubelet/pods/0c3ff369-203e-45ad-9fb3-adc3bffd3f72/volumes"
Jul 27 20:35:53.568117 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:53.568054    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d72d783b-c69e-4a88-be14-0ce7f23ec3a5" path="/var/lib/kubelet/pods/d72d783b-c69e-4a88-be14-0ce7f23ec3a5/volumes"
Jul 27 20:35:56.253846 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:56.253818    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:56.257273 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:56.257244    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"]
Jul 27 20:35:56.259426 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:56.259394    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d68d13-z699p"]
Jul 27 20:35:56.653661 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:56.653618    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:35:57.567825 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:35:57.567796    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd41c27-d3ec-4e43-ae53-3f2eac44dc21" path="/var/lib/kubelet/pods/3fd41c27-d3ec-4e43-ae53-3f2eac44dc21/volumes"
Jul 27 20:36:03.152887 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:03.152798    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:03.255453 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:03.255419    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:03.409691 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:03.409605    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:03.635672 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:03.635640    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:14.432680 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.432650    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"]
Jul 27 20:36:14.433231 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.432860    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" podUID="5a0da973-1c0c-4e94-b833-2b06e68a117b" containerName="maas-api" containerID="cri-o://c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635" gracePeriod=30
Jul 27 20:36:14.460492 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.460463    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"]
Jul 27 20:36:14.461392 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.461363    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:14.462963 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.462943    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-a78f44-29753070-jz9bn"]
Jul 27 20:36:14.491626 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.491586    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"]
Jul 27 20:36:14.491847 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.491820    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk" podUID="09afcc06-b883-4b85-b631-dd0eb9007d76" containerName="payload-processing" containerID="cri-o://83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473" gracePeriod=30
Jul 27 20:36:14.497359 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.497330    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"]
Jul 27 20:36:14.497595 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.497566    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2" podUID="181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" containerName="payload-pre-processing" containerID="cri-o://93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2" gracePeriod=30
Jul 27 20:36:14.530735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.530713    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:14.606548 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.606489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:14.753936 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.753907    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:36:14.797108 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.797082    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:36:14.803935 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.803915    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:36:14.810296 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.810280    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09afcc06-b883-4b85-b631-dd0eb9007d76-plugins-config-volume\") pod \"09afcc06-b883-4b85-b631-dd0eb9007d76\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") "
Jul 27 20:36:14.810413 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.810401    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdn4q\" (UniqueName: \"kubernetes.io/projected/09afcc06-b883-4b85-b631-dd0eb9007d76-kube-api-access-rdn4q\") pod \"09afcc06-b883-4b85-b631-dd0eb9007d76\" (UID: \"09afcc06-b883-4b85-b631-dd0eb9007d76\") "
Jul 27 20:36:14.810735 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.810702    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09afcc06-b883-4b85-b631-dd0eb9007d76-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09afcc06-b883-4b85-b631-dd0eb9007d76" (UID: "09afcc06-b883-4b85-b631-dd0eb9007d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:14.812579 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.812554    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09afcc06-b883-4b85-b631-dd0eb9007d76-kube-api-access-rdn4q" (OuterVolumeSpecName: "kube-api-access-rdn4q") pod "09afcc06-b883-4b85-b631-dd0eb9007d76" (UID: "09afcc06-b883-4b85-b631-dd0eb9007d76"). InnerVolumeSpecName "kube-api-access-rdn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:14.911612 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911577    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-kube-api-access-8zrn4\") pod \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") "
Jul 27 20:36:14.911612 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911613    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-plugins-config-volume\") pod \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\" (UID: \"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7\") "
Jul 27 20:36:14.911872 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911646    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drlxm\" (UniqueName: \"kubernetes.io/projected/5a0da973-1c0c-4e94-b833-2b06e68a117b-kube-api-access-drlxm\") pod \"5a0da973-1c0c-4e94-b833-2b06e68a117b\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") "
Jul 27 20:36:14.911872 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911686    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0da973-1c0c-4e94-b833-2b06e68a117b-openshift-service-ca\") pod \"5a0da973-1c0c-4e94-b833-2b06e68a117b\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") "
Jul 27 20:36:14.911872 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911713    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls\") pod \"5a0da973-1c0c-4e94-b833-2b06e68a117b\" (UID: \"5a0da973-1c0c-4e94-b833-2b06e68a117b\") "
Jul 27 20:36:14.912038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911978    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdn4q\" (UniqueName: \"kubernetes.io/projected/09afcc06-b883-4b85-b631-dd0eb9007d76-kube-api-access-rdn4q\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:14.912038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.911999    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09afcc06-b883-4b85-b631-dd0eb9007d76-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:14.912038 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.912013    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" (UID: "181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:14.912187 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.912093    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0da973-1c0c-4e94-b833-2b06e68a117b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a0da973-1c0c-4e94-b833-2b06e68a117b" (UID: "5a0da973-1c0c-4e94-b833-2b06e68a117b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:14.913808 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.913788    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-kube-api-access-8zrn4" (OuterVolumeSpecName: "kube-api-access-8zrn4") pod "181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" (UID: "181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7"). InnerVolumeSpecName "kube-api-access-8zrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:14.913926 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.913900    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0da973-1c0c-4e94-b833-2b06e68a117b-kube-api-access-drlxm" (OuterVolumeSpecName: "kube-api-access-drlxm") pod "5a0da973-1c0c-4e94-b833-2b06e68a117b" (UID: "5a0da973-1c0c-4e94-b833-2b06e68a117b"). InnerVolumeSpecName "kube-api-access-drlxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:14.913969 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:14.913911    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a0da973-1c0c-4e94-b833-2b06e68a117b" (UID: "5a0da973-1c0c-4e94-b833-2b06e68a117b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:15.012803 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.012715    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrn4\" (UniqueName: \"kubernetes.io/projected/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-kube-api-access-8zrn4\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:15.012803 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.012747    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:15.012803 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.012757    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlxm\" (UniqueName: \"kubernetes.io/projected/5a0da973-1c0c-4e94-b833-2b06e68a117b-kube-api-access-drlxm\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:15.012803 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.012765    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0da973-1c0c-4e94-b833-2b06e68a117b-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:15.012803 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.012774    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0da973-1c0c-4e94-b833-2b06e68a117b-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:15.082588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.082550    2568 generic.go:356] "Generic (PLEG): container finished" podID="5a0da973-1c0c-4e94-b833-2b06e68a117b" containerID="c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635" exitCode=0
Jul 27 20:36:15.082588 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.082589    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"
Jul 27 20:36:15.082820 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.082624    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" event={"ID":"5a0da973-1c0c-4e94-b833-2b06e68a117b","Type":"ContainerDied","Data":"c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635"}
Jul 27 20:36:15.082820 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.082684    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s" event={"ID":"5a0da973-1c0c-4e94-b833-2b06e68a117b","Type":"ContainerDied","Data":"7fcb4ba7f21bb0d8b1571b027e6f6caa8f4c68b05997644816068554f69e5708"}
Jul 27 20:36:15.082820 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.082716    2568 scope.go:117] "RemoveContainer" containerID="c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635"
Jul 27 20:36:15.084091 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.084069    2568 generic.go:356] "Generic (PLEG): container finished" podID="181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" containerID="93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2" exitCode=0
Jul 27 20:36:15.084194 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.084105    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"
Jul 27 20:36:15.084194 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.084145    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2" event={"ID":"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7","Type":"ContainerDied","Data":"93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2"}
Jul 27 20:36:15.084296 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.084198    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2" event={"ID":"181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7","Type":"ContainerDied","Data":"5655fe56d5002b0395f85185c3a911080b18b7be6ff1eca0874675035f4b7dd0"}
Jul 27 20:36:15.085867 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.085846    2568 generic.go:356] "Generic (PLEG): container finished" podID="09afcc06-b883-4b85-b631-dd0eb9007d76" containerID="83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473" exitCode=0
Jul 27 20:36:15.085978 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.085895    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"
Jul 27 20:36:15.086045 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.085895    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk" event={"ID":"09afcc06-b883-4b85-b631-dd0eb9007d76","Type":"ContainerDied","Data":"83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473"}
Jul 27 20:36:15.086045 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.086024    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk" event={"ID":"09afcc06-b883-4b85-b631-dd0eb9007d76","Type":"ContainerDied","Data":"001c21b9097705558c2ece92138cde14260cc0f4c824afc0b61ec6e199671cda"}
Jul 27 20:36:15.093483 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.093460    2568 scope.go:117] "RemoveContainer" containerID="c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635"
Jul 27 20:36:15.093807 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:36:15.093782    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635\": container with ID starting with c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635 not found: ID does not exist" containerID="c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635"
Jul 27 20:36:15.093884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.093819    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635"} err="failed to get container status \"c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635\": rpc error: code = NotFound desc = could not find container \"c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635\": container with ID starting with c7c92b59d84d40d0f3d2e44436fc57bd1ee1cf25496de044b9759a1f53149635 not found: ID does not exist"
Jul 27 20:36:15.093884 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.093843    2568 scope.go:117] "RemoveContainer" containerID="93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2"
Jul 27 20:36:15.102344 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.102326    2568 scope.go:117] "RemoveContainer" containerID="93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2"
Jul 27 20:36:15.102652 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:36:15.102633    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2\": container with ID starting with 93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2 not found: ID does not exist" containerID="93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2"
Jul 27 20:36:15.102731 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.102658    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2"} err="failed to get container status \"93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2\": rpc error: code = NotFound desc = could not find container \"93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2\": container with ID starting with 93be8773c7b9b83d6fda3f0d838ac134b9f3db81b1cadbce4ea84ebd25d1c6f2 not found: ID does not exist"
Jul 27 20:36:15.102731 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.102675    2568 scope.go:117] "RemoveContainer" containerID="83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473"
Jul 27 20:36:15.110136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.110117    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"]
Jul 27 20:36:15.111096 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.111078    2568 scope.go:117] "RemoveContainer" containerID="83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473"
Jul 27 20:36:15.111364 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:36:15.111344    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473\": container with ID starting with 83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473 not found: ID does not exist" containerID="83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473"
Jul 27 20:36:15.111463 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.111368    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473"} err="failed to get container status \"83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473\": rpc error: code = NotFound desc = could not find container \"83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473\": container with ID starting with 83e4db242bf8cc81cf077709ae6b64f16d09a0596517317469b113f2cfb97473 not found: ID does not exist"
Jul 27 20:36:15.114309 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.114288    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a78f44-5c4b44df76-v89g2"]
Jul 27 20:36:15.125794 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.125775    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"]
Jul 27 20:36:15.131377 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.131356    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a78f44-775b9889cf-zrh9s"]
Jul 27 20:36:15.143633 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.143613    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"]
Jul 27 20:36:15.145645 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.145626    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a78f44-5cb479758-rq8fk"]
Jul 27 20:36:15.567383 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.567352    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09afcc06-b883-4b85-b631-dd0eb9007d76" path="/var/lib/kubelet/pods/09afcc06-b883-4b85-b631-dd0eb9007d76/volumes"
Jul 27 20:36:15.567776 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.567748    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" path="/var/lib/kubelet/pods/181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7/volumes"
Jul 27 20:36:15.568089 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.568075    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0da973-1c0c-4e94-b833-2b06e68a117b" path="/var/lib/kubelet/pods/5a0da973-1c0c-4e94-b833-2b06e68a117b/volumes"
Jul 27 20:36:15.568309 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:15.568298    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4575446-4609-4b9a-ba93-057e93e5b2ec" path="/var/lib/kubelet/pods/b4575446-4609-4b9a-ba93-057e93e5b2ec/volumes"
Jul 27 20:36:18.901909 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:18.901874    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:19.280007 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:19.279983    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:23.845797 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:23.845762    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:26.575836 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:26.575804    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:26.676623 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:26.676589    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:26.845932 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:26.845840    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:27.066611 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.066574    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:27.170264 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170186    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"]
Jul 27 20:36:27.170583 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170568    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72d783b-c69e-4a88-be14-0ce7f23ec3a5" containerName="payload-pre-processing"
Jul 27 20:36:27.170583 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170583    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72d783b-c69e-4a88-be14-0ce7f23ec3a5" containerName="payload-pre-processing"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170593    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd41c27-d3ec-4e43-ae53-3f2eac44dc21" containerName="revoke-keys"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170599    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd41c27-d3ec-4e43-ae53-3f2eac44dc21" containerName="revoke-keys"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170614    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" containerName="payload-pre-processing"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170620    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" containerName="payload-pre-processing"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170627    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09afcc06-b883-4b85-b631-dd0eb9007d76" containerName="payload-processing"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170633    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="09afcc06-b883-4b85-b631-dd0eb9007d76" containerName="payload-processing"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170640    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3ff369-203e-45ad-9fb3-adc3bffd3f72" containerName="maas-api"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170645    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3ff369-203e-45ad-9fb3-adc3bffd3f72" containerName="maas-api"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170658    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0da973-1c0c-4e94-b833-2b06e68a117b" containerName="maas-api"
Jul 27 20:36:27.170670 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170663    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0da973-1c0c-4e94-b833-2b06e68a117b" containerName="maas-api"
Jul 27 20:36:27.170970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170722    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0da973-1c0c-4e94-b833-2b06e68a117b" containerName="maas-api"
Jul 27 20:36:27.170970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170731    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd41c27-d3ec-4e43-ae53-3f2eac44dc21" containerName="revoke-keys"
Jul 27 20:36:27.170970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170738    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="09afcc06-b883-4b85-b631-dd0eb9007d76" containerName="payload-processing"
Jul 27 20:36:27.170970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170745    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72d783b-c69e-4a88-be14-0ce7f23ec3a5" containerName="payload-pre-processing"
Jul 27 20:36:27.170970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170751    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="181a332a-b79e-4d9b-b9ca-daf8d2c9cbd7" containerName="payload-pre-processing"
Jul 27 20:36:27.170970 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.170757    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3ff369-203e-45ad-9fb3-adc3bffd3f72" containerName="maas-api"
Jul 27 20:36:27.175315 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.175299    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:27.178188 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.178167    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pljl4\""
Jul 27 20:36:27.181994 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.181971    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"]
Jul 27 20:36:27.228123 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.228086    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8db4484-9144-402e-9eaf-f3824354aa95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6\" (UID: \"a8db4484-9144-402e-9eaf-f3824354aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:27.328930 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.328892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8db4484-9144-402e-9eaf-f3824354aa95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6\" (UID: \"a8db4484-9144-402e-9eaf-f3824354aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:27.329576 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.329559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8db4484-9144-402e-9eaf-f3824354aa95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6\" (UID: \"a8db4484-9144-402e-9eaf-f3824354aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:27.488491 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.488411    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:27.631421 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:27.631396    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"]
Jul 27 20:36:27.634076 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:36:27.634048    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8db4484_9144_402e_9eaf_f3824354aa95.slice/crio-4ac7b4dffcdac6a651d77a7f57a509c33f7fd5dcb8d7ef2a36d16b879e96d6b4 WatchSource:0}: Error finding container 4ac7b4dffcdac6a651d77a7f57a509c33f7fd5dcb8d7ef2a36d16b879e96d6b4: Status 404 returned error can't find the container with id 4ac7b4dffcdac6a651d77a7f57a509c33f7fd5dcb8d7ef2a36d16b879e96d6b4
Jul 27 20:36:28.155139 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:28.155054    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6" event={"ID":"a8db4484-9144-402e-9eaf-f3824354aa95","Type":"ContainerStarted","Data":"8ed1d12e79a8c5559b739c444ab6112edcd85953d2ba078a78ba8e730bbce059"}
Jul 27 20:36:28.155139 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:28.155103    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6" event={"ID":"a8db4484-9144-402e-9eaf-f3824354aa95","Type":"ContainerStarted","Data":"4ac7b4dffcdac6a651d77a7f57a509c33f7fd5dcb8d7ef2a36d16b879e96d6b4"}
Jul 27 20:36:28.174139 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:28.174067    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6" podStartSLOduration=1.174047631 podStartE2EDuration="1.174047631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:36:28.17386223 +0000 UTC m=+4337.169710210" watchObservedRunningTime="2026-07-27 20:36:28.174047631 +0000 UTC m=+4337.169895610"
Jul 27 20:36:29.161846 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:29.161811    2568 generic.go:356] "Generic (PLEG): container finished" podID="a8db4484-9144-402e-9eaf-f3824354aa95" containerID="8ed1d12e79a8c5559b739c444ab6112edcd85953d2ba078a78ba8e730bbce059" exitCode=0
Jul 27 20:36:29.162286 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:29.161873    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6" event={"ID":"a8db4484-9144-402e-9eaf-f3824354aa95","Type":"ContainerDied","Data":"8ed1d12e79a8c5559b739c444ab6112edcd85953d2ba078a78ba8e730bbce059"}
Jul 27 20:36:30.306875 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:30.300507    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:30.361263 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:30.361231    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8db4484-9144-402e-9eaf-f3824354aa95-maas-api-service-ca\") pod \"a8db4484-9144-402e-9eaf-f3824354aa95\" (UID: \"a8db4484-9144-402e-9eaf-f3824354aa95\") "
Jul 27 20:36:30.361664 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:30.361637    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8db4484-9144-402e-9eaf-f3824354aa95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8db4484-9144-402e-9eaf-f3824354aa95" (UID: "a8db4484-9144-402e-9eaf-f3824354aa95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:30.462680 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:30.462599    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8db4484-9144-402e-9eaf-f3824354aa95-maas-api-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:31.174181 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:31.174152    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"
Jul 27 20:36:31.174181 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:31.174168    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6" event={"ID":"a8db4484-9144-402e-9eaf-f3824354aa95","Type":"ContainerDied","Data":"4ac7b4dffcdac6a651d77a7f57a509c33f7fd5dcb8d7ef2a36d16b879e96d6b4"}
Jul 27 20:36:31.174398 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:31.174211    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ac7b4dffcdac6a651d77a7f57a509c33f7fd5dcb8d7ef2a36d16b879e96d6b4"
Jul 27 20:36:37.836946 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.836902    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"]
Jul 27 20:36:37.837310 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.837147    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" podUID="9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" containerName="maas-api" containerID="cri-o://5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79" gracePeriod=30
Jul 27 20:36:37.851679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.851649    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:37.885672 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.885639    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"]
Jul 27 20:36:37.885903 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.885876    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6" podUID="d339a66c-c034-46c1-ae27-cc4110066d0d" containerName="payload-processing" containerID="cri-o://6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018" gracePeriod=30
Jul 27 20:36:37.894699 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.894659    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"]
Jul 27 20:36:37.894944 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.894890    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk" podUID="a339ace1-88ea-441f-8306-daad94f35ec4" containerName="payload-pre-processing" containerID="cri-o://d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56" gracePeriod=30
Jul 27 20:36:37.926725 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.926697    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:37.983802 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:37.983776    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:38.175331 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.175305    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:36:38.178572 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.178549    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:36:38.182073 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.182056    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:36:38.219296 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.219261    2568 generic.go:356] "Generic (PLEG): container finished" podID="d339a66c-c034-46c1-ae27-cc4110066d0d" containerID="6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018" exitCode=0
Jul 27 20:36:38.219476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.219304    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"
Jul 27 20:36:38.219476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.219350    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6" event={"ID":"d339a66c-c034-46c1-ae27-cc4110066d0d","Type":"ContainerDied","Data":"6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018"}
Jul 27 20:36:38.219476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.219422    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6" event={"ID":"d339a66c-c034-46c1-ae27-cc4110066d0d","Type":"ContainerDied","Data":"c3b49f986358c0388225d726d33f3459bdecf72d87dd144d46be76db0ecf9705"}
Jul 27 20:36:38.219476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.219461    2568 scope.go:117] "RemoveContainer" containerID="6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018"
Jul 27 20:36:38.220754 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.220736    2568 generic.go:356] "Generic (PLEG): container finished" podID="9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" containerID="5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79" exitCode=0
Jul 27 20:36:38.220846 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.220790    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"
Jul 27 20:36:38.220846 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.220812    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" event={"ID":"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b","Type":"ContainerDied","Data":"5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79"}
Jul 27 20:36:38.220963 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.220878    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2" event={"ID":"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b","Type":"ContainerDied","Data":"912ce7c902744a9da5582d869179071f6013fd3c44d17fbd93e65148fa0eea51"}
Jul 27 20:36:38.222341 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.222321    2568 generic.go:356] "Generic (PLEG): container finished" podID="a339ace1-88ea-441f-8306-daad94f35ec4" containerID="d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56" exitCode=0
Jul 27 20:36:38.222445 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.222368    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk" event={"ID":"a339ace1-88ea-441f-8306-daad94f35ec4","Type":"ContainerDied","Data":"d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56"}
Jul 27 20:36:38.222445 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.222424    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk" event={"ID":"a339ace1-88ea-441f-8306-daad94f35ec4","Type":"ContainerDied","Data":"22b2ce2aa35c6f52e2f5d53ff47b3fafb85d9e0d59821a843c2b894eb82a4bae"}
Jul 27 20:36:38.222557 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.222372    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"
Jul 27 20:36:38.230082 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.230060    2568 scope.go:117] "RemoveContainer" containerID="6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018"
Jul 27 20:36:38.230472 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:36:38.230434    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018\": container with ID starting with 6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018 not found: ID does not exist" containerID="6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018"
Jul 27 20:36:38.230600 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.230480    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018"} err="failed to get container status \"6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018\": rpc error: code = NotFound desc = could not find container \"6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018\": container with ID starting with 6ea74f457014f24a57067393680bc6d9ad5549920bfc6b15587d5b1487792018 not found: ID does not exist"
Jul 27 20:36:38.230600 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.230504    2568 scope.go:117] "RemoveContainer" containerID="5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79"
Jul 27 20:36:38.239762 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.239743    2568 scope.go:117] "RemoveContainer" containerID="5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79"
Jul 27 20:36:38.240054 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:36:38.240033    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79\": container with ID starting with 5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79 not found: ID does not exist" containerID="5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79"
Jul 27 20:36:38.240116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.240065    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79"} err="failed to get container status \"5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79\": rpc error: code = NotFound desc = could not find container \"5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79\": container with ID starting with 5c4fde15400e9ec35fac2b2442727e18ea2bbf55ce26c6a80e193086d1688c79 not found: ID does not exist"
Jul 27 20:36:38.240116 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.240087    2568 scope.go:117] "RemoveContainer" containerID="d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56"
Jul 27 20:36:38.248357 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.248337    2568 scope.go:117] "RemoveContainer" containerID="d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56"
Jul 27 20:36:38.248678 ip-10-0-143-3 kubenswrapper[2568]: E0727 20:36:38.248658    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56\": container with ID starting with d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56 not found: ID does not exist" containerID="d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56"
Jul 27 20:36:38.248750 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.248688    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56"} err="failed to get container status \"d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56\": rpc error: code = NotFound desc = could not find container \"d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56\": container with ID starting with d075603b29a437c4cff187265b73ca59d36c1c2fa71747ff347cd4981f79ba56 not found: ID does not exist"
Jul 27 20:36:38.340951 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.340919    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d339a66c-c034-46c1-ae27-cc4110066d0d-plugins-config-volume\") pod \"d339a66c-c034-46c1-ae27-cc4110066d0d\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") "
Jul 27 20:36:38.341136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.340979    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwcqz\" (UniqueName: \"kubernetes.io/projected/a339ace1-88ea-441f-8306-daad94f35ec4-kube-api-access-bwcqz\") pod \"a339ace1-88ea-441f-8306-daad94f35ec4\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") "
Jul 27 20:36:38.341136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.340997    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls\") pod \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") "
Jul 27 20:36:38.341136 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341032    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-openshift-service-ca\") pod \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") "
Jul 27 20:36:38.341369 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341137    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47bps\" (UniqueName: \"kubernetes.io/projected/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-kube-api-access-47bps\") pod \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\" (UID: \"9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b\") "
Jul 27 20:36:38.341369 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341191    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcsc4\" (UniqueName: \"kubernetes.io/projected/d339a66c-c034-46c1-ae27-cc4110066d0d-kube-api-access-tcsc4\") pod \"d339a66c-c034-46c1-ae27-cc4110066d0d\" (UID: \"d339a66c-c034-46c1-ae27-cc4110066d0d\") "
Jul 27 20:36:38.341369 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341304    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339ace1-88ea-441f-8306-daad94f35ec4-plugins-config-volume\") pod \"a339ace1-88ea-441f-8306-daad94f35ec4\" (UID: \"a339ace1-88ea-441f-8306-daad94f35ec4\") "
Jul 27 20:36:38.341519 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341379    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d339a66c-c034-46c1-ae27-cc4110066d0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d339a66c-c034-46c1-ae27-cc4110066d0d" (UID: "d339a66c-c034-46c1-ae27-cc4110066d0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.341519 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341387    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" (UID: "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.341779 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341724    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a339ace1-88ea-441f-8306-daad94f35ec4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a339ace1-88ea-441f-8306-daad94f35ec4" (UID: "a339ace1-88ea-441f-8306-daad94f35ec4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:36:38.341779 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341779    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-openshift-service-ca\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.341779 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.341798    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d339a66c-c034-46c1-ae27-cc4110066d0d-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.343391 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.343330    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a339ace1-88ea-441f-8306-daad94f35ec4-kube-api-access-bwcqz" (OuterVolumeSpecName: "kube-api-access-bwcqz") pod "a339ace1-88ea-441f-8306-daad94f35ec4" (UID: "a339ace1-88ea-441f-8306-daad94f35ec4"). InnerVolumeSpecName "kube-api-access-bwcqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:38.343391 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.343360    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d339a66c-c034-46c1-ae27-cc4110066d0d-kube-api-access-tcsc4" (OuterVolumeSpecName: "kube-api-access-tcsc4") pod "d339a66c-c034-46c1-ae27-cc4110066d0d" (UID: "d339a66c-c034-46c1-ae27-cc4110066d0d"). InnerVolumeSpecName "kube-api-access-tcsc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:38.343806 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.343777    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" (UID: "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:36:38.343806 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.343790    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-kube-api-access-47bps" (OuterVolumeSpecName: "kube-api-access-47bps") pod "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" (UID: "9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b"). InnerVolumeSpecName "kube-api-access-47bps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:36:38.442495 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.442458    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47bps\" (UniqueName: \"kubernetes.io/projected/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-kube-api-access-47bps\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.442495 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.442490    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcsc4\" (UniqueName: \"kubernetes.io/projected/d339a66c-c034-46c1-ae27-cc4110066d0d-kube-api-access-tcsc4\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.442495 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.442498    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a339ace1-88ea-441f-8306-daad94f35ec4-plugins-config-volume\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.442752 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.442507    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwcqz\" (UniqueName: \"kubernetes.io/projected/a339ace1-88ea-441f-8306-daad94f35ec4-kube-api-access-bwcqz\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.442752 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.442519    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b-maas-api-tls\") on node \"ip-10-0-143-3.ec2.internal\" DevicePath \"\""
Jul 27 20:36:38.549225 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.549195    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"]
Jul 27 20:36:38.554641 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.554617    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-43d589-f489964f4-mgdn6"]
Jul 27 20:36:38.565598 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.565564    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"]
Jul 27 20:36:38.569855 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.569832    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-43d589-c9f99b96-6zqrk"]
Jul 27 20:36:38.580933 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.580909    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"]
Jul 27 20:36:38.583603 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:38.583584    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-43d589-cd6f68695-jnzz2"]
Jul 27 20:36:39.567759 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:39.567727    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" path="/var/lib/kubelet/pods/9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b/volumes"
Jul 27 20:36:39.568143 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:39.568097    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a339ace1-88ea-441f-8306-daad94f35ec4" path="/var/lib/kubelet/pods/a339ace1-88ea-441f-8306-daad94f35ec4/volumes"
Jul 27 20:36:39.568402 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:39.568389    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d339a66c-c034-46c1-ae27-cc4110066d0d" path="/var/lib/kubelet/pods/d339a66c-c034-46c1-ae27-cc4110066d0d/volumes"
Jul 27 20:36:42.246207 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:42.246176    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:42.247684 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:42.247644    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"]
Jul 27 20:36:42.249328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:42.249308    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-43d589-zhtm6"]
Jul 27 20:36:42.623848 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:42.623820    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:43.567594 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:43.567567    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8db4484-9144-402e-9eaf-f3824354aa95" path="/var/lib/kubelet/pods/a8db4484-9144-402e-9eaf-f3824354aa95/volumes"
Jul 27 20:36:49.236198 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:49.236154    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:49.336427 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:49.336396    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:49.491671 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:49.491589    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:49.713711 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:49.713678    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:36:55.852924 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:55.852894    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5c9bc998-5kxml_a9816b64-77d3-4fb4-8314-982231604886/copy-manifests/0.log"
Jul 27 20:36:55.859056 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:55.859026    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5c9bc998-5kxml_a9816b64-77d3-4fb4-8314-982231604886/manager/0.log"
Jul 27 20:36:56.437973 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:56.437942    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84cf8569bc-4wtjd_e96386aa-a353-4998-b7a1-15b1b8c38496/maas-api/0.log"
Jul 27 20:36:56.544060 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:56.544028    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753040-ct4kr_2ea61ffc-a6ab-4553-a5ce-98c939a6840d/cleanup/0.log"
Jul 27 20:36:56.652417 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:56.652389    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753055-f864b_9f108ba5-c3c7-458d-ad2a-98af57b83616/cleanup/0.log"
Jul 27 20:36:56.758727 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:56.758655    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753070-9pd44_a323b8e3-0cf3-4985-8a48-779815d2856b/cleanup/0.log"
Jul 27 20:36:56.864106 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:56.864081    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vcjnk_d3723976-c34f-41ba-9b19-83d330df2ce7/postgres/0.log"
Jul 27 20:36:58.329231 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:58.329191    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8qbnf_91fcffbe-6d2a-4e46-8aed-f47a152f550e/manager/0.log"
Jul 27 20:36:58.438690 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:58.438658    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7x6j2_1fe84938-3a52-4844-8351-46474577827f/kuadrant-console-plugin/0.log"
Jul 27 20:36:59.537990 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:59.537961    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bbdd9d5dd-tbrch_7f02614d-835b-4b11-b9a8-22f371c89f23/kube-auth-proxy/0.log"
Jul 27 20:36:59.877310 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:36:59.877231    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dsffp_a5b8f564-3897-4a48-aaa0-4faa317ca28d/payload-processing/0.log"
Jul 27 20:37:00.424009 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:00.423983    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xggcf_3ececa3a-ca84-4e35-aa82-6386ec88a69b/main/0.log"
Jul 27 20:37:05.811621 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811582    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9vsps/must-gather-dvfhr"]
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811945    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8db4484-9144-402e-9eaf-f3824354aa95" containerName="revoke-keys"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811956    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8db4484-9144-402e-9eaf-f3824354aa95" containerName="revoke-keys"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811965    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" containerName="maas-api"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811971    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" containerName="maas-api"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811985    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d339a66c-c034-46c1-ae27-cc4110066d0d" containerName="payload-processing"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.811990    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d339a66c-c034-46c1-ae27-cc4110066d0d" containerName="payload-processing"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.812000    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a339ace1-88ea-441f-8306-daad94f35ec4" containerName="payload-pre-processing"
Jul 27 20:37:05.812012 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.812005    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a339ace1-88ea-441f-8306-daad94f35ec4" containerName="payload-pre-processing"
Jul 27 20:37:05.812263 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.812056    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d339a66c-c034-46c1-ae27-cc4110066d0d" containerName="payload-processing"
Jul 27 20:37:05.812263 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.812066    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bcd6d6e-9c96-4d26-ad0c-3575d3cb6a8b" containerName="maas-api"
Jul 27 20:37:05.812263 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.812075    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a339ace1-88ea-441f-8306-daad94f35ec4" containerName="payload-pre-processing"
Jul 27 20:37:05.812263 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.812081    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8db4484-9144-402e-9eaf-f3824354aa95" containerName="revoke-keys"
Jul 27 20:37:05.815295 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.815275    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:05.815416 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.815372    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:37:05.817963 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.817943    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vsps\"/\"openshift-service-ca.crt\""
Jul 27 20:37:05.819125 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.819107    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9vsps\"/\"default-dockercfg-gfl9g\""
Jul 27 20:37:05.819208 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.819107    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vsps\"/\"kube-root-ca.crt\""
Jul 27 20:37:05.823681 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.823658    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vsps/must-gather-dvfhr"]
Jul 27 20:37:05.882010 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.881972    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/569d675d-da01-4c40-98da-8d0dc48a57c4-must-gather-output\") pod \"must-gather-dvfhr\" (UID: \"569d675d-da01-4c40-98da-8d0dc48a57c4\") " pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:05.882196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.882096    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89p6\" (UniqueName: \"kubernetes.io/projected/569d675d-da01-4c40-98da-8d0dc48a57c4-kube-api-access-k89p6\") pod \"must-gather-dvfhr\" (UID: \"569d675d-da01-4c40-98da-8d0dc48a57c4\") " pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:05.915401 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.915369    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:37:05.983618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.983579    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89p6\" (UniqueName: \"kubernetes.io/projected/569d675d-da01-4c40-98da-8d0dc48a57c4-kube-api-access-k89p6\") pod \"must-gather-dvfhr\" (UID: \"569d675d-da01-4c40-98da-8d0dc48a57c4\") " pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:05.983801 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.983638    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/569d675d-da01-4c40-98da-8d0dc48a57c4-must-gather-output\") pod \"must-gather-dvfhr\" (UID: \"569d675d-da01-4c40-98da-8d0dc48a57c4\") " pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:05.983972 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.983956    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/569d675d-da01-4c40-98da-8d0dc48a57c4-must-gather-output\") pod \"must-gather-dvfhr\" (UID: \"569d675d-da01-4c40-98da-8d0dc48a57c4\") " pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:05.992758 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:05.992731    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89p6\" (UniqueName: \"kubernetes.io/projected/569d675d-da01-4c40-98da-8d0dc48a57c4-kube-api-access-k89p6\") pod \"must-gather-dvfhr\" (UID: \"569d675d-da01-4c40-98da-8d0dc48a57c4\") " pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:06.125059 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:06.124971    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsps/must-gather-dvfhr"
Jul 27 20:37:06.266161 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:06.266132    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vsps/must-gather-dvfhr"]
Jul 27 20:37:06.268837 ip-10-0-143-3 kubenswrapper[2568]: W0727 20:37:06.268810    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod569d675d_da01_4c40_98da_8d0dc48a57c4.slice/crio-a183854e3690146054debbfb01ba1df9feb240a6b4f961a31095b1258d94a5f7 WatchSource:0}: Error finding container a183854e3690146054debbfb01ba1df9feb240a6b4f961a31095b1258d94a5f7: Status 404 returned error can't find the container with id a183854e3690146054debbfb01ba1df9feb240a6b4f961a31095b1258d94a5f7
Jul 27 20:37:06.270731 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:06.270716    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:37:06.367440 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:06.367404    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsps/must-gather-dvfhr" event={"ID":"569d675d-da01-4c40-98da-8d0dc48a57c4","Type":"ContainerStarted","Data":"a183854e3690146054debbfb01ba1df9feb240a6b4f961a31095b1258d94a5f7"}
Jul 27 20:37:07.378711 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:07.378663    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsps/must-gather-dvfhr" event={"ID":"569d675d-da01-4c40-98da-8d0dc48a57c4","Type":"ContainerStarted","Data":"2a503547491fa6b4c72595b6f6ca44ad69916f34d7e759df48d117db22878633"}
Jul 27 20:37:07.379222 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:07.378729    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsps/must-gather-dvfhr" event={"ID":"569d675d-da01-4c40-98da-8d0dc48a57c4","Type":"ContainerStarted","Data":"8fa88b939bb7e9074be6cb181f08f6726100d95383cfae2a8c800dea63f9c119"}
Jul 27 20:37:07.399911 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:07.399836    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9vsps/must-gather-dvfhr" podStartSLOduration=1.65184686 podStartE2EDuration="2.399814112s" totalImagesPullingTime="747.967252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:07.39718113 +0000 UTC m=+4376.393029129" watchObservedRunningTime="2026-07-27 20:37:07.399814112 +0000 UTC m=+4376.395662091"
Jul 27 20:37:08.686723 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:08.686680    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x49q7_047be226-fd26-402a-99b4-ebc0dadca574/global-pull-secret-syncer/0.log"
Jul 27 20:37:08.740099 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:08.740063    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log"
Jul 27 20:37:08.894618 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:08.894581    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-3.ec2.internal_6180e30e1977ed4d8d23fa6e6c33c1ed/haproxy/0.log"
Jul 27 20:37:10.201258 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:10.201210    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:37:10.265141 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:10.265104    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qmp5w_5f592725-be92-4809-a830-e569fabe7890/konnectivity-agent/0.log" limit=1024
Jul 27 20:37:12.204412 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:12.204381    2568 scope.go:117] "RemoveContainer" containerID="e5f0b91e7df3cb5191580146a274153ccacacadd15c99b60f626fdd339b9139b"
Jul 27 20:37:14.012268 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:14.012171    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8qbnf_91fcffbe-6d2a-4e46-8aed-f47a152f550e/manager/0.log"
Jul 27 20:37:14.032705 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:14.032664    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7x6j2_1fe84938-3a52-4844-8351-46474577827f/kuadrant-console-plugin/0.log"
Jul 27 20:37:15.960491 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:15.960454    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-lbh6d_8fd5bbbe-6712-4160-a12a-b318f7e0387c/cluster-monitoring-operator/0.log"
Jul 27 20:37:16.133621 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.133593    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77m7m_4c30b188-012b-4ed4-8908-768f6fbb7e2a/node-exporter/0.log"
Jul 27 20:37:16.168927 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.168902    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77m7m_4c30b188-012b-4ed4-8908-768f6fbb7e2a/kube-rbac-proxy/0.log"
Jul 27 20:37:16.193328 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.193304    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77m7m_4c30b188-012b-4ed4-8908-768f6fbb7e2a/init-textfile/0.log"
Jul 27 20:37:16.491099 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.491011    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/prometheus/0.log"
Jul 27 20:37:16.513653 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.513619    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/config-reloader/0.log"
Jul 27 20:37:16.535196 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.535170    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/thanos-sidecar/0.log"
Jul 27 20:37:16.559542 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.559493    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/kube-rbac-proxy-web/0.log"
Jul 27 20:37:16.581017 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.580985    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/kube-rbac-proxy/0.log"
Jul 27 20:37:16.603679 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.603646    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/kube-rbac-proxy-thanos/0.log"
Jul 27 20:37:16.624078 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.624048    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c25cc069-254a-49db-966b-cf666ef8932f/init-config-reloader/0.log"
Jul 27 20:37:16.822576 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.822542    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d4f79c7-rgckd_e12a874c-f838-42cd-98db-2936008d6f86/thanos-query/0.log"
Jul 27 20:37:16.845682 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.845651    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d4f79c7-rgckd_e12a874c-f838-42cd-98db-2936008d6f86/kube-rbac-proxy-web/0.log"
Jul 27 20:37:16.866537 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.866496    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d4f79c7-rgckd_e12a874c-f838-42cd-98db-2936008d6f86/kube-rbac-proxy/0.log"
Jul 27 20:37:16.887148 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.887121    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d4f79c7-rgckd_e12a874c-f838-42cd-98db-2936008d6f86/prom-label-proxy/0.log"
Jul 27 20:37:16.908511 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.908482    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d4f79c7-rgckd_e12a874c-f838-42cd-98db-2936008d6f86/kube-rbac-proxy-rules/0.log"
Jul 27 20:37:16.933145 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:16.933120    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-849d4f79c7-rgckd_e12a874c-f838-42cd-98db-2936008d6f86/kube-rbac-proxy-metrics/0.log"
Jul 27 20:37:18.915655 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:18.915622    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"]
Jul 27 20:37:18.916700 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:18.916664    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/2.log"
Jul 27 20:37:18.921388 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:18.921363    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:18.922972 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:18.922952    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-459pm_0bcb2391-77ca-4b27-854a-e63d209d7def/console-operator/3.log"
Jul 27 20:37:18.929570 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:18.929509    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"]
Jul 27 20:37:19.026546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.025107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmkf\" (UniqueName: \"kubernetes.io/projected/1250e8dc-8c90-4106-adab-59614a8af862-kube-api-access-wlmkf\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.026546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.025186    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-proc\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.026546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.025273    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-lib-modules\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.026546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.025332    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-sys\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.026546 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.025371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-podres\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126282 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-podres\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126322    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmkf\" (UniqueName: \"kubernetes.io/projected/1250e8dc-8c90-4106-adab-59614a8af862-kube-api-access-wlmkf\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-proc\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126422    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-podres\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126476 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-lib-modules\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126694 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-sys\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126694 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126548    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-proc\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126694 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126569    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-lib-modules\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.126694 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.126630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1250e8dc-8c90-4106-adab-59614a8af862-sys\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.135709 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.135677    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmkf\" (UniqueName: \"kubernetes.io/projected/1250e8dc-8c90-4106-adab-59614a8af862-kube-api-access-wlmkf\") pod \"perf-node-gather-daemonset-wnddk\" (UID: \"1250e8dc-8c90-4106-adab-59614a8af862\") " pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.238018 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.237941    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:19.427783 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.427728    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"]
Jul 27 20:37:19.467999 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.467962    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk" event={"ID":"1250e8dc-8c90-4106-adab-59614a8af862","Type":"ContainerStarted","Data":"728af031799f9a1cf6320749fae24fda199d023cf0a545f6037eb3525b2bd061"}
Jul 27 20:37:19.541481 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.541452    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7954c54d9b-vbmzh_1b73b0c8-3b97-4d78-a170-5e084f5be495/console/0.log"
Jul 27 20:37:19.593085 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:19.592980    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-pzckr_edefbf77-cf8c-4499-abe5-65ee93dae38e/download-server/0.log"
Jul 27 20:37:20.256063 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:20.256028    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-sgscc_c4ae37b7-a0e5-4d21-af44-03c07bdc6e5d/volume-data-source-validator/0.log"
Jul 27 20:37:20.476994 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:20.476957    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk" event={"ID":"1250e8dc-8c90-4106-adab-59614a8af862","Type":"ContainerStarted","Data":"54ffc407428ea6f3b603454d54827a5bdad0f45b4fbb462b6b00279cda994049"}
Jul 27 20:37:20.477193 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:20.477072    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:20.495373 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:20.495301    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk" podStartSLOduration=2.495278032 podStartE2EDuration="2.495278032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:37:20.492926642 +0000 UTC m=+4389.488774617" watchObservedRunningTime="2026-07-27 20:37:20.495278032 +0000 UTC m=+4389.491126011"
Jul 27 20:37:21.424660 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:21.424635    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kkx66_a6852432-508f-4243-84ef-f9aebd2e2ae3/dns/0.log"
Jul 27 20:37:21.445105 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:21.445073    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kkx66_a6852432-508f-4243-84ef-f9aebd2e2ae3/kube-rbac-proxy/0.log"
Jul 27 20:37:21.559539 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:21.559497    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-slvtt_40900963-6214-4d92-8e58-7f7ab9fff63a/dns-node-resolver/0.log"
Jul 27 20:37:22.105749 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:22.105713    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69f478d788-xt6q7_dbdd8b60-768e-4803-92fa-c2bae4520a55/registry/0.log"
Jul 27 20:37:22.150378 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:22.150351    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcp6x_61ab69d2-8341-4561-8ed0-9873bff1eec1/node-ca/0.log"
Jul 27 20:37:23.569244 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:23.569210    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bbdd9d5dd-tbrch_7f02614d-835b-4b11-b9a8-22f371c89f23/kube-auth-proxy/0.log"
Jul 27 20:37:23.681693 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:23.681660    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dsffp_a5b8f564-3897-4a48-aaa0-4faa317ca28d/payload-processing/0.log"
Jul 27 20:37:24.583778 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:24.583718    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xrhc7_39c9cba3-d810-4681-b777-a3bb5abad710/serve-healthcheck-canary/0.log"
Jul 27 20:37:25.233465 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:25.233439    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mbvtm_75395a11-454f-4e19-bd8d-842ccda55d1d/kube-rbac-proxy/0.log"
Jul 27 20:37:25.252846 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:25.252818    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mbvtm_75395a11-454f-4e19-bd8d-842ccda55d1d/exporter/0.log"
Jul 27 20:37:25.273091 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:25.273065    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mbvtm_75395a11-454f-4e19-bd8d-842ccda55d1d/extractor/0.log"
Jul 27 20:37:26.492597 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:26.492568    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9vsps/perf-node-gather-daemonset-wnddk"
Jul 27 20:37:28.121064 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:28.121025    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5c9bc998-5kxml_a9816b64-77d3-4fb4-8314-982231604886/manager/0.log"
Jul 27 20:37:28.145115 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:28.145091    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5c9bc998-5kxml_a9816b64-77d3-4fb4-8314-982231604886/copy-manifests/0.log"
Jul 27 20:37:29.643925 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:29.643890    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-776dc7b77-44wlv_f715dda3-a4ee-4004-8c48-396b41d0321b/manager/0.log"
Jul 27 20:37:29.871317 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:29.871288    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bbdd9d5dd-tbrch_7f02614d-835b-4b11-b9a8-22f371c89f23/kube-auth-proxy/0.log"
Jul 27 20:37:29.962303 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:29.962238    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dsffp_a5b8f564-3897-4a48-aaa0-4faa317ca28d/payload-processing/0.log"
Jul 27 20:37:31.604694 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:31.604661    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bbdd9d5dd-tbrch_7f02614d-835b-4b11-b9a8-22f371c89f23/kube-auth-proxy/0.log"
Jul 27 20:37:31.696096 ip-10-0-143-3 kubenswrapper[2568]: I0727 20:37:31.696062    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-dsffp_a5b8f564-3897-4a48-aaa0-4faa317ca28d/payload-processing/0.log"