Aug 04 12:25:39.056521 ip-10-0-136-46 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 12:25:39.056534 ip-10-0-136-46 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 12:25:39.056543 ip-10-0-136-46 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 12:25:39.056858 ip-10-0-136-46 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 12:25:49.305618 ip-10-0-136-46 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 12:25:49.305646 ip-10-0-136-46 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 065608f0236645fbb35a3865402de777 -- Aug 04 12:27:47.754478 ip-10-0-136-46 systemd[1]: Starting Kubernetes Kubelet... Aug 04 12:27:48.114755 ip-10-0-136-46 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 12:27:48.114755 ip-10-0-136-46 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 12:27:48.114755 ip-10-0-136-46 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 12:27:48.114755 ip-10-0-136-46 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 12:27:48.114755 ip-10-0-136-46 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 12:27:48.115849 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.115711 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 12:27:48.119979 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.119962 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.119985 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.119989 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.119992 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.119995 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.119998 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120003 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120008 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120012 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120016 2578 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120019 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120022 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120025 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120028 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 12:27:48.120024 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120031 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120034 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120037 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120039 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120044 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120047 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120050 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120055 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120057 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120060 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120063 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120065 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120068 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120070 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120073 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120076 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120078 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120089 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120093 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120095 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 12:27:48.120414 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120098 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120100 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120117 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120120 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120123 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120125 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120128 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120130 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120133 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120135 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120138 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120140 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120143 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120145 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120148 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120151 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120154 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120156 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120159 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 12:27:48.120887 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120161 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120164 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120167 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120169 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120171 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120174 2578 feature_gate.go:328] unrecognized feature gate: Example Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120176 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120179 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120181 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120184 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120186 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120189 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120192 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120195 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120197 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120199 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120202 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120204 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120206 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120209 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 12:27:48.121361 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120211 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120214 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120216 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120218 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120221 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120223 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120226 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120230 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120233 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120236 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120238 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120241 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120243 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120246 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120248 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120250 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120253 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120255 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120258 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120260 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 12:27:48.121839 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120262 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120264 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120267 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120270 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120273 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.120275 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121005 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121013 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121016 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121019 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121022 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121024 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121027 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121029 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121032 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121034 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121037 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121039 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121042 2578 feature_gate.go:328] unrecognized feature gate: Example Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121045 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 12:27:48.122329 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121048 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121051 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121054 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121058 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121062 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121065 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121069 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121072 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121074 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121077 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121079 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121082 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121084 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121087 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121089 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121092 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121095 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121098 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121101 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121120 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 12:27:48.122847 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121123 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121126 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121129 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121131 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121134 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121136 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121139 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121142 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121145 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121147 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121150 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121153 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121155 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121158 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121160 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121163 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121166 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121168 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121172 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121175 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 12:27:48.123346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121177 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121179 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121182 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121184 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121187 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121189 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121191 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121194 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121196 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121199 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121202 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121205 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121207 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121209 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121212 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121214 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121217 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121219 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121224 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 12:27:48.123825 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121228 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121232 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121236 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121239 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121245 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121248 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121250 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121253 2578 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121256 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121258 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121260 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121263 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121265 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121268 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121270 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121273 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121275 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121278 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121280 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 12:27:48.124346 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121283 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121286 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121288 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121291 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121293 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121296 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121299 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121670 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121675 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121679 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121681 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121684 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121688 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121691 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121693 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121696 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121698 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121700 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121703 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121705 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 12:27:48.124801 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121708 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121710 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121713 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121716 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121718 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121721 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121724 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121728 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121731 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121734 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121738 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121741 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121744 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121746 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121749 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121752 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121754 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121758 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121761 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 12:27:48.125298 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121763 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121767 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121770 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121772 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121775 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121779 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121782 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121784 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121787 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121789 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121792 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121794 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121797 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121800 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121802 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121806 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121808 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121811 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121813 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 12:27:48.125750 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121816 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121819 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121821 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121824 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121826 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121829 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121831 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121834 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121836 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121840 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121843 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121846 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121850 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121852 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121857 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121860 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121862 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121865 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121870 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 12:27:48.126228 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121872 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121875 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121877 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121880 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121882 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121885 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121887 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121890 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121892 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121895 2578 feature_gate.go:328] unrecognized feature gate: Example Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121897 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121900 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121902 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121904 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121907 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121909 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121912 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121914 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121917 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121919 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 12:27:48.126687 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121922 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121924 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121927 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121929 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121931 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121934 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121940 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121944 2578 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.121947 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122020 2578 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122028 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122037 2578 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122043 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122049 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122054 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122059 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122066 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122070 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122073 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122076 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122080 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122082 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 12:27:48.127259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122085 2578 flags.go:64] FLAG: --cgroup-root="" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122088 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122091 2578 flags.go:64] FLAG: --client-ca-file="" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122094 2578 flags.go:64] FLAG: --cloud-provider="external" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122096 2578 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122100 2578 flags.go:64] FLAG: --cluster-domain="" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122115 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122119 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122122 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122126 2578 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122129 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122132 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122135 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122139 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122142 2578 flags.go:64] FLAG: --contention-profiling="false" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122144 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122147 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122153 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122156 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122160 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122163 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122167 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122170 2578 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122172 2578 flags.go:64] FLAG: --enable-server="true" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122175 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 12:27:48.127812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122180 2578 flags.go:64] FLAG: --event-burst="100" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122183 2578 flags.go:64] FLAG: --event-qps="50" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122186 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122190 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122193 2578 flags.go:64] FLAG: --eviction-hard="" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122196 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122199 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122202 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122204 2578 flags.go:64] FLAG: --eviction-soft="" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122207 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122210 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122213 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122215 2578 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122218 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122221 2578 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122224 2578 flags.go:64] FLAG: --feature-gates="" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122227 2578 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122230 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122233 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122236 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122239 2578 flags.go:64] FLAG: --healthz-port="10248" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122242 2578 flags.go:64] FLAG: --help="false" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122245 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122248 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 12:27:48.128418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122252 2578 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122256 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122259 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122263 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122266 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122269 2578 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122272 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122275 2578 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122277 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122280 2578 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122283 2578 flags.go:64] FLAG: --kube-reserved="" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122286 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122289 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122292 2578 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122294 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122297 2578 flags.go:64] FLAG: --lock-file="" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122300 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122303 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122306 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122311 2578 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122314 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122317 2578 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122319 2578 flags.go:64] FLAG: --logging-format="text" Aug 04 12:27:48.129080 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122322 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122325 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122328 2578 flags.go:64] FLAG: --manifest-url="" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122331 2578 flags.go:64] FLAG: --manifest-url-header="" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122335 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122338 2578 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122342 2578 flags.go:64] FLAG: --max-pods="110" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122346 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122348 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122353 2578 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122356 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122359 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122362 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122365 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122372 2578 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122375 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122378 2578 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122381 2578 flags.go:64] FLAG: --pod-cidr="" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122384 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122389 2578 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122392 2578 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122395 2578 flags.go:64] FLAG: --pods-per-core="0" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122398 2578 flags.go:64] FLAG: --port="10250" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122401 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 12:27:48.129658 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122404 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bf26ef877902c144" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122407 2578 flags.go:64] FLAG: --qos-reserved="" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122409 2578 flags.go:64] FLAG: --read-only-port="10255" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122412 2578 flags.go:64] FLAG: --register-node="true" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122415 2578 flags.go:64] FLAG: --register-with-taints="" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122418 2578 flags.go:64] FLAG: --registry-burst="10" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122421 2578 flags.go:64] FLAG: --registry-qps="5" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122423 2578 flags.go:64] FLAG: --reserved-cpus="" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122426 2578 flags.go:64] FLAG: --reserved-memory="" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122430 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122432 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122435 2578 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122438 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122440 2578 flags.go:64] FLAG: --runonce="false" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122443 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122446 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122449 2578 flags.go:64] FLAG: --seccomp-default="false" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122452 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122455 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122458 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122462 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122464 2578 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122467 2578 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122470 2578 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122473 2578 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122476 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 12:27:48.130242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122479 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122482 2578 flags.go:64] FLAG: --system-cgroups="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122484 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122489 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122492 2578 flags.go:64] FLAG: --tls-cert-file="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122495 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122498 2578 flags.go:64] FLAG: --tls-min-version="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122501 2578 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122504 2578 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122507 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122510 2578 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122513 2578 flags.go:64] FLAG: --v="2" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122517 2578 flags.go:64] FLAG: --version="false" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122521 2578 flags.go:64] FLAG: --vmodule="" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122525 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.122528 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122664 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122668 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122672 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122675 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122677 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122680 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122683 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 12:27:48.130914 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122688 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122693 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122696 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122698 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122702 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122704 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122707 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122709 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122712 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122714 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122717 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122719 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122722 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122724 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122726 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122729 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122732 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122735 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122737 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 12:27:48.131837 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122740 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122742 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122744 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122748 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122751 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122754 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122757 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122759 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122762 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122765 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122767 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122769 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122772 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122776 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122779 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122781 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122783 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122786 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122789 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 12:27:48.132400 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122791 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122794 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122796 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122799 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122801 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122803 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122806 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122808 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122810 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122813 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122815 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122817 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122820 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122822 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122825 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122827 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122830 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122832 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122835 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122837 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 12:27:48.132905 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122839 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122842 2578 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122844 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122846 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122849 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122852 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122855 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122858 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122861 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122863 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122865 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122869 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122872 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122875 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122877 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122880 2578 feature_gate.go:328] unrecognized feature gate: Example Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122882 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122884 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122887 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122889 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122892 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 12:27:48.133428 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122894 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122896 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122899 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122901 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122905 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122908 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122911 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122913 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122916 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122919 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122921 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122924 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.122926 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.123527 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.131771 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 12:27:48.133955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.131794 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.131821 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.131830 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132040 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132204 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132210 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132214 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132219 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132224 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132227 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132230 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132233 2578 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132236 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132240 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132243 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132247 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132250 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132253 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132256 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 12:27:48.134342 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132259 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132261 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132264 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132267 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132269 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132272 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132275 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132278 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132280 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132283 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132286 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132288 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132291 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132299 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132302 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132305 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132307 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132310 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132313 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132315 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 12:27:48.134802 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132318 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132321 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132323 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132326 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132328 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132331 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132333 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132336 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132339 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132341 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132343 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132346 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132348 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132351 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132354 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132356 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132359 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132361 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132364 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132366 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 12:27:48.135289 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132369 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132371 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132373 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132376 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132378 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132381 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132384 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132386 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132389 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132391 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132394 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132396 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132399 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132401 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132404 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132406 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132409 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132412 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132415 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132418 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 12:27:48.135770 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132421 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132423 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132425 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132428 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132431 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132433 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132437 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132440 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132443 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132446 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132448 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132451 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132453 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132455 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132458 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132461 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132464 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132468 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132470 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132473 2578 feature_gate.go:328] unrecognized feature gate: Example Aug 04 12:27:48.136363 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132476 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132479 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132481 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.132487 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132617 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132623 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132626 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132629 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132632 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132635 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132638 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132640 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132643 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132645 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132648 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 12:27:48.136893 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132650 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132660 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132663 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132665 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132668 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132670 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132673 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132676 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132678 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132681 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132684 2578 feature_gate.go:328] unrecognized feature gate: Example Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132686 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132689 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132692 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132694 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132697 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132700 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132702 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132705 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132708 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 12:27:48.137321 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132710 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132713 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132715 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132718 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132720 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132723 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132726 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132728 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132731 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132733 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132736 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132738 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132740 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132743 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132747 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132751 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132754 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132757 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132760 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132762 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 12:27:48.137929 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132765 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132768 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132770 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132773 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132775 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132778 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132781 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132784 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132786 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132789 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132791 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132793 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132796 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132799 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132801 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132804 2578 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132806 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132809 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132811 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132813 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 12:27:48.138430 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132816 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132818 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132821 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132824 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132827 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132831 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132833 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132836 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132839 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132841 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132844 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132846 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132849 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132851 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132854 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132856 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132859 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132861 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132864 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 12:27:48.139125 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132866 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132868 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132871 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132873 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132876 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132878 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132880 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132883 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:48.132887 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.132892 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.133018 2578 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.136203 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.137420 2578 server.go:1019] "Starting client certificate rotation" Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.137540 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 12:27:48.139886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.137631 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 12:27:48.140269 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.139899 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 12:27:48.141151 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.141128 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 12:27:48.150493 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.150476 2578 log.go:25] "Validated CRI v1 runtime API" Aug 04 12:27:48.154573 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.154559 2578 log.go:25] "Validated CRI v1 image API" Aug 04 12:27:48.155465 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.155451 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 12:27:48.157628 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.157602 2578 fs.go:135] Filesystem UUIDs: map[5fa1f8d6-fc74-4dca-9a76-74a68c613b69:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d24236d6-9753-4da2-96a0-1b4d5857da12:/dev/nvme0n1p3] Aug 04 12:27:48.157693 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.157627 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 12:27:48.162587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.162478 2578 manager.go:216] Machine: {Timestamp:2026-08-04 12:27:48.161893986 +0000 UTC m=+0.299504159 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100180 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23eb69692c209ab3e03ce48ac92e13 SystemUUID:ec23eb69-692c-209a-b3e0-3ce48ac92e13 BootID:065608f0-2366-45fb-b35a-3865402de777 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a3:37:f6:95:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a3:37:f6:95:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:65:d1:93:10:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 12:27:48.162587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.162583 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 12:27:48.162700 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.162660 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 12:27:48.165567 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.165542 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 12:27:48.165723 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.165569 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-46.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 12:27:48.166273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.166263 2578 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 12:27:48.166313 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.166275 2578 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 12:27:48.166313 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.166283 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 12:27:48.166855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.166845 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 12:27:48.167401 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.167385 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 12:27:48.167595 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.167585 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 12:27:48.171664 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171653 2578 state_mem.go:36] "Initialized new in-memory state store" Aug 04 12:27:48.171789 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171779 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 12:27:48.171826 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171813 2578 kubelet.go:486] "Attempting to sync node with API server" Aug 04 12:27:48.171857 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171835 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 12:27:48.171886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171863 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 12:27:48.171886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171878 2578 kubelet.go:398] "Adding apiserver pod source" Aug 04 12:27:48.171945 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.171905 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 12:27:48.173807 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.173742 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 12:27:48.173917 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.173840 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 12:27:48.177166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.177147 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 12:27:48.178387 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.178374 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 12:27:48.178427 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.178405 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 12:27:48.179335 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179324 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 12:27:48.179366 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179340 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 12:27:48.179366 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179347 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 12:27:48.179366 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179352 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 12:27:48.179366 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179360 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 12:27:48.179366 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179365 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179373 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179379 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179385 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179391 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179399 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179404 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179428 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 12:27:48.179501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.179434 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 12:27:48.182910 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.182893 2578 server.go:1268] "Started kubelet" Aug 04 12:27:48.183081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.182991 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 12:27:48.183172 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.183127 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 12:27:48.183426 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.183009 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 12:27:48.183848 ip-10-0-136-46 systemd[1]: Started Kubernetes Kubelet. Aug 04 12:27:48.184731 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.184715 2578 server.go:310] "Adding debug handlers to kubelet server" Aug 04 12:27:48.185029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.184947 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 12:27:48.185864 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.185844 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-46.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 12:27:48.186250 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.186227 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-46.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 12:27:48.186296 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.186228 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 12:27:48.188414 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.188388 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 12:27:48.188967 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.188951 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 12:27:48.189025 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189008 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 12:27:48.189410 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.189290 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.189759 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189734 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 12:27:48.189759 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189750 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 12:27:48.189891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189752 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 12:27:48.189891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189848 2578 factory.go:55] Registering systemd factory Aug 04 12:27:48.189891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189865 2578 factory.go:223] Registration of the systemd container factory successfully Aug 04 12:27:48.189891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189881 2578 reconstruct.go:97] "Volume reconstruction finished" Aug 04 12:27:48.189891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.189889 2578 reconciler.go:29] "Reconciler: start to sync state" Aug 04 12:27:48.190196 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.190177 2578 factory.go:153] Registering CRI-O factory Aug 04 12:27:48.190196 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.190199 2578 factory.go:223] Registration of the crio container factory successfully Aug 04 12:27:48.190325 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.190252 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 12:27:48.190325 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.189258 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-46.ec2.internal.18c899e6a76cf6d2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-46.ec2.internal,UID:ip-10-0-136-46.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-46.ec2.internal,},FirstTimestamp:2026-08-04 12:27:48.182865618 +0000 UTC m=+0.320475791,LastTimestamp:2026-08-04 12:27:48.182865618 +0000 UTC m=+0.320475791,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-46.ec2.internal,}" Aug 04 12:27:48.190325 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.190292 2578 factory.go:103] Registering Raw factory Aug 04 12:27:48.190325 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.190307 2578 manager.go:1196] Started watching for new ooms in manager Aug 04 12:27:48.190911 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.190898 2578 manager.go:349] Starting recovery of all containers Aug 04 12:27:48.191801 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.191778 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9gw5r" Aug 04 12:27:48.197231 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.197200 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 12:27:48.197231 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.197211 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-46.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 12:27:48.201953 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.201935 2578 manager.go:354] Recovery completed Aug 04 12:27:48.202392 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.202277 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9gw5r" Aug 04 12:27:48.206980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.206964 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.208914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.208901 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.208980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.208927 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.208980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.208937 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.209502 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.209489 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 12:27:48.209502 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.209501 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 12:27:48.209575 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.209507 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.209575 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.209517 2578 state_mem.go:36] "Initialized new in-memory state store" Aug 04 12:27:48.210194 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.210177 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.210273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.210210 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.210273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.210224 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.211148 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.211058 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-46.ec2.internal.18c899e6a8fa729e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-46.ec2.internal,UID:ip-10-0-136-46.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-46.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-46.ec2.internal,},FirstTimestamp:2026-08-04 12:27:48.208915102 +0000 UTC m=+0.346525278,LastTimestamp:2026-08-04 12:27:48.208915102 +0000 UTC m=+0.346525278,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-46.ec2.internal,}" Aug 04 12:27:48.211597 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.211586 2578 policy_none.go:49] "None policy: Start" Aug 04 12:27:48.211631 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.211612 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 12:27:48.211631 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.211623 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 12:27:48.213603 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.213590 2578 policy_none.go:47] "Start" Aug 04 12:27:48.245749 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.245727 2578 manager.go:337] "Starting Device Plugin manager" Aug 04 12:27:48.245864 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.245762 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 12:27:48.245864 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.245771 2578 server.go:87] "Starting device plugin registration server" Aug 04 12:27:48.246303 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.246289 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 12:27:48.246377 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.246312 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 12:27:48.246479 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.246459 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 12:27:48.246579 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.246566 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 12:27:48.246655 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.246579 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 12:27:48.247050 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.247031 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 12:27:48.247141 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.247077 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.346538 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.346506 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.348740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.347500 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.348740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.347538 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.348740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.347552 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.348740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.347577 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.355272 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.355248 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.355272 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.355273 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-46.ec2.internal\": node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.363051 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.363018 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 12:27:48.364536 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.364517 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 12:27:48.364634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.364553 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 12:27:48.364634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.364576 2578 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 12:27:48.364634 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.364610 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 12:27:48.366308 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.366251 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 12:27:48.385884 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.385858 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.465773 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.465719 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal"] Aug 04 12:27:48.465936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.465815 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.467369 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.467349 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.467491 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.467380 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.467491 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.467392 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.469855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.469840 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.469979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.469962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.470026 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.469998 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.470647 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.470630 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.470740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.470648 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.470740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.470661 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.470740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.470667 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.470740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.470674 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.470740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.470676 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.473024 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.473007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.473100 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.473037 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 12:27:48.476182 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.476159 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientMemory" Aug 04 12:27:48.476254 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.476198 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 12:27:48.476304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.476256 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeHasSufficientPID" Aug 04 12:27:48.485938 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.485910 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.486713 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.486693 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-46.ec2.internal\" not found" node="ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.491101 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.491064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b5d07ef5da4396d8fc84d8570a1cf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal\" (UID: \"55b5d07ef5da4396d8fc84d8570a1cf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.491205 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.491127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b5d07ef5da4396d8fc84d8570a1cf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal\" (UID: \"55b5d07ef5da4396d8fc84d8570a1cf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.491205 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.491156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a2944b9797870016f92974beffaabc3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.ec2.internal\" (UID: \"1a2944b9797870016f92974beffaabc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.492289 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.492268 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-46.ec2.internal\" not found" node="ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.586346 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.586315 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.591692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.591671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b5d07ef5da4396d8fc84d8570a1cf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal\" (UID: \"55b5d07ef5da4396d8fc84d8570a1cf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.591775 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.591702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a2944b9797870016f92974beffaabc3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.ec2.internal\" (UID: \"1a2944b9797870016f92974beffaabc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.591775 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.591722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b5d07ef5da4396d8fc84d8570a1cf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal\" (UID: \"55b5d07ef5da4396d8fc84d8570a1cf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.591775 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.591769 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b5d07ef5da4396d8fc84d8570a1cf0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal\" (UID: \"55b5d07ef5da4396d8fc84d8570a1cf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.591885 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.591781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b5d07ef5da4396d8fc84d8570a1cf0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal\" (UID: \"55b5d07ef5da4396d8fc84d8570a1cf0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.591885 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.591788 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a2944b9797870016f92974beffaabc3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.ec2.internal\" (UID: \"1a2944b9797870016f92974beffaabc3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.687448 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.687377 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.787944 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.787905 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.789022 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.789004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.795325 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:48.795301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" Aug 04 12:27:48.889092 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.889049 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:48.989605 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:48.989512 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:49.089953 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:49.089918 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:50.319325 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.319266 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 12:27:50.319842 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.319509 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 12:27:50.372045 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:50.372011 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:50.389202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.389160 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 12:22:49 +0000 UTC" deadline="2028-02-23 07:48:31.881050259 +0000 UTC" Aug 04 12:27:50.389202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.389192 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13627h20m41.491864742s" Aug 04 12:27:50.424993 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.424918 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 12:27:50.473025 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:50.472992 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:50.573362 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:50.573329 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-46.ec2.internal\" not found" Aug 04 12:27:50.608772 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.608422 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 12:27:50.666406 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.666357 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 12:27:50.671710 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.671692 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" Aug 04 12:27:50.684736 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.684649 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 12:27:50.685512 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.685496 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" Aug 04 12:27:50.695383 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.695364 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 12:27:50.739292 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:50.739251 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2944b9797870016f92974beffaabc3.slice/crio-8a0ef364d260a5cb58c5cefeec81c08630f085d37b7c4fc1e9dc9726b8bd5807 WatchSource:0}: Error finding container 8a0ef364d260a5cb58c5cefeec81c08630f085d37b7c4fc1e9dc9726b8bd5807: Status 404 returned error can't find the container with id 8a0ef364d260a5cb58c5cefeec81c08630f085d37b7c4fc1e9dc9726b8bd5807 Aug 04 12:27:50.741205 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:50.741151 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b5d07ef5da4396d8fc84d8570a1cf0.slice/crio-565cfb9fb82bc90ca65f63d5c277719a5a045096d79e53911ba4ba0efca9ba91 WatchSource:0}: Error finding container 565cfb9fb82bc90ca65f63d5c277719a5a045096d79e53911ba4ba0efca9ba91: Status 404 returned error can't find the container with id 565cfb9fb82bc90ca65f63d5c277719a5a045096d79e53911ba4ba0efca9ba91 Aug 04 12:27:50.746240 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:50.746218 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 12:27:51.354855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.354826 2578 apiserver.go:52] "Watching apiserver" Aug 04 12:27:51.363668 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.363640 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 12:27:51.364030 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.363999 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kd7rr","kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jqglq","openshift-dns/node-resolver-7vggh","openshift-image-registry/node-ca-9dwhz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal","openshift-multus/multus-bc27s","openshift-multus/network-metrics-daemon-z28dh","kube-system/konnectivity-agent-8j5qs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp","openshift-multus/multus-additional-cni-plugins-tlnnv","openshift-network-diagnostics/network-check-target-46dcw","openshift-network-operator/iptables-alerter-7qzvv","openshift-ovn-kubernetes/ovnkube-node-qch7k"] Aug 04 12:27:51.366564 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.366538 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.368661 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.368638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.369209 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.369186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 12:27:51.369672 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.369650 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.369777 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.369709 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-88sf5\"" Aug 04 12:27:51.369848 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.369832 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.369905 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.369872 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 12:27:51.371142 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.371098 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.371502 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.371484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.372408 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.372387 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.372507 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.372409 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 12:27:51.372507 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.372433 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fvwcn\"" Aug 04 12:27:51.372639 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.372623 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 12:27:51.374426 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.374223 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.374426 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.374275 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.374571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.374481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.374571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.374501 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bzl6k\"" Aug 04 12:27:51.376647 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.376629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.377886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.377865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.377969 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.377924 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q7ddl\"" Aug 04 12:27:51.378011 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.377872 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.378998 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.378975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.379340 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.379323 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.379340 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.379332 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 12:27:51.379464 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.379394 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6czjj\"" Aug 04 12:27:51.379523 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.379506 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.381308 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5460bc2-716e-485b-a943-b65d58b0bb32-tmp\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktppd\" (UniqueName: \"kubernetes.io/projected/b5460bc2-716e-485b-a943-b65d58b0bb32-kube-api-access-ktppd\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc89e5a3-43c5-4413-85ce-67c1df2eb916-cni-binary-copy\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-etc-kubernetes\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-lib-modules\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-os-release\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.381412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-k8s-cni-cncf-io\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-cni-multus\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-hostroot\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-host-slash\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysctl-d\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-run\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-sys\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3b799a-900a-4bff-be86-986adc410241-tmp-dir\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-socket-dir-parent\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-netns\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-conf-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r2w\" (UniqueName: \"kubernetes.io/projected/cc89e5a3-43c5-4413-85ce-67c1df2eb916-kube-api-access-n2r2w\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-systemd\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-host\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-multus-certs\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.381738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-iptables-alerter-script\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-modprobe-d\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-cni-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-cni-bin\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-kubelet\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysctl-conf\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-var-lib-kubelet\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-tuned\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f3b799a-900a-4bff-be86-986adc410241-hosts-file\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjbl\" (UniqueName: \"kubernetes.io/projected/8f3b799a-900a-4bff-be86-986adc410241-kube-api-access-qvjbl\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.381985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-daemon-config\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-kubernetes\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-system-cni-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-cnibin\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vnr8\" (UniqueName: \"kubernetes.io/projected/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-kube-api-access-4vnr8\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382266 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382317 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382416 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 12:27:51.382527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysconfig\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.383375 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.382567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 12:27:51.383375 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.383257 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 12:27:51.383375 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.383268 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bgvl4\"" Aug 04 12:27:51.383375 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.383276 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 12:27:51.384007 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.383985 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 12:27:51.384268 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.384248 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n9t8g\"" Aug 04 12:27:51.384487 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.384467 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 12:27:51.384891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.384729 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 12:27:51.384891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.384766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.384891 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.384827 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:27:51.387571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.387555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:51.387662 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.387641 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:27:51.389407 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.389385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.392267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.392245 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 12:27:51.392424 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.392403 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l6stk\"" Aug 04 12:27:51.392521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.392500 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 12:27:51.392583 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.392521 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 12:27:51.392733 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.392718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.394599 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.394578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:51.394673 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.394644 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:27:51.395207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.395187 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s5g82\"" Aug 04 12:27:51.395412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.395246 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 12:27:51.395412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.395254 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 12:27:51.408412 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.408392 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xqdds" Aug 04 12:27:51.416756 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.416736 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xqdds" Aug 04 12:27:51.472796 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.472725 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 12:27:51.483646 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vnr8\" (UniqueName: \"kubernetes.io/projected/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-kube-api-access-4vnr8\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktppd\" (UniqueName: \"kubernetes.io/projected/b5460bc2-716e-485b-a943-b65d58b0bb32-kube-api-access-ktppd\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc89e5a3-43c5-4413-85ce-67c1df2eb916-cni-binary-copy\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-etc-kubernetes\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn6d8\" (UniqueName: \"kubernetes.io/projected/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-kube-api-access-mn6d8\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-k8s-cni-cncf-io\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-hostroot\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.483825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-os-release\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnvd\" (UniqueName: \"kubernetes.io/projected/564ce3c2-fbd1-400a-993d-7310abe7a910-kube-api-access-8xnvd\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-ovnkube-config\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ljn\" (UniqueName: \"kubernetes.io/projected/fc7df39c-e1e8-45a5-a852-a75592bd193e-kube-api-access-q4ljn\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-host-slash\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.483987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysctl-d\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-sys\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-etc-kubernetes\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3b799a-900a-4bff-be86-986adc410241-tmp-dir\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-socket-dir-parent\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484067 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-k8s-cni-cncf-io\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-systemd-units\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484096 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-hostroot\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r2w\" (UniqueName: \"kubernetes.io/projected/cc89e5a3-43c5-4413-85ce-67c1df2eb916-kube-api-access-n2r2w\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.484235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-kubelet\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysctl-d\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ca687e-554b-41b0-9feb-40167b647220-ovn-node-metrics-cert\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d82696e-2556-495c-93d3-4b882156c2a4-agent-certs\") pod \"konnectivity-agent-8j5qs\" (UID: \"7d82696e-2556-495c-93d3-4b882156c2a4\") " pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484355 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-host-slash\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484408 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-sys\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5xd\" (UniqueName: \"kubernetes.io/projected/4fc0096c-c114-448c-ac13-639b117d40f3-kube-api-access-rp5xd\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-cni-binary-copy\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-ovn\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-log-socket\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-modprobe-d\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc89e5a3-43c5-4413-85ce-67c1df2eb916-cni-binary-copy\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-cni-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-cni-bin\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484614 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-socket-dir-parent\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-kubelet\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484643 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-cni-bin\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-modprobe-d\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484712 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-kubelet\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484698 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f3b799a-900a-4bff-be86-986adc410241-tmp-dir\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-dbus\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484786 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-cni-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysctl-conf\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-var-lib-kubelet\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-tuned\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f3b799a-900a-4bff-be86-986adc410241-hosts-file\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f3b799a-900a-4bff-be86-986adc410241-hosts-file\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysctl-conf\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.484990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-var-lib-kubelet\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjbl\" (UniqueName: \"kubernetes.io/projected/8f3b799a-900a-4bff-be86-986adc410241-kube-api-access-qvjbl\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-slash\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-etc-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-kubernetes\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485148 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 12:27:51.485823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-system-cni-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-cnibin\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-run-ovn-kubernetes\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-cni-bin\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-ovnkube-script-lib\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysconfig\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485417 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-sysconfig\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485426 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-cnibin\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-kubernetes\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5460bc2-716e-485b-a943-b65d58b0bb32-tmp\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-system-cni-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-cnibin\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255tp\" (UniqueName: \"kubernetes.io/projected/72ca687e-554b-41b0-9feb-40167b647220-kube-api-access-255tp\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-device-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.486689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-lib-modules\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-os-release\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-cni-multus\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-system-cni-dir\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-cni-netd\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485814 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-var-lib-cni-multus\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc7df39c-e1e8-45a5-a852-a75592bd193e-host\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-os-release\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-run\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-env-overrides\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc7df39c-e1e8-45a5-a852-a75592bd193e-serviceca\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-lib-modules\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.485998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-kubelet-config\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-run\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-netns\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486132 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-netns\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.487519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-conf-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-conf-dir\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-systemd\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-host\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-multus-certs\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486413 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-systemd\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-node-log\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5460bc2-716e-485b-a943-b65d58b0bb32-host\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc89e5a3-43c5-4413-85ce-67c1df2eb916-host-run-multus-certs\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-iptables-alerter-script\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-var-lib-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-daemon-config\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-systemd\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-run-netns\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.488381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.486963 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d82696e-2556-495c-93d3-4b882156c2a4-konnectivity-ca\") pod \"konnectivity-agent-8j5qs\" (UID: \"7d82696e-2556-495c-93d3-4b882156c2a4\") " pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.489049 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.487065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-iptables-alerter-script\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.489049 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.487593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc89e5a3-43c5-4413-85ce-67c1df2eb916-multus-daemon-config\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.490166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.489954 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5460bc2-716e-485b-a943-b65d58b0bb32-tmp\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.490166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.490106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5460bc2-716e-485b-a943-b65d58b0bb32-etc-tuned\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.495610 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.495570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktppd\" (UniqueName: \"kubernetes.io/projected/b5460bc2-716e-485b-a943-b65d58b0bb32-kube-api-access-ktppd\") pod \"tuned-jqglq\" (UID: \"b5460bc2-716e-485b-a943-b65d58b0bb32\") " pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.495731 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.495694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r2w\" (UniqueName: \"kubernetes.io/projected/cc89e5a3-43c5-4413-85ce-67c1df2eb916-kube-api-access-n2r2w\") pod \"multus-bc27s\" (UID: \"cc89e5a3-43c5-4413-85ce-67c1df2eb916\") " pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.496850 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.496826 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjbl\" (UniqueName: \"kubernetes.io/projected/8f3b799a-900a-4bff-be86-986adc410241-kube-api-access-qvjbl\") pod \"node-resolver-7vggh\" (UID: \"8f3b799a-900a-4bff-be86-986adc410241\") " pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.497198 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.497178 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vnr8\" (UniqueName: \"kubernetes.io/projected/7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea-kube-api-access-4vnr8\") pod \"iptables-alerter-7qzvv\" (UID: \"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea\") " pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.552240 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.552178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" event={"ID":"1a2944b9797870016f92974beffaabc3","Type":"ContainerStarted","Data":"8a0ef364d260a5cb58c5cefeec81c08630f085d37b7c4fc1e9dc9726b8bd5807"} Aug 04 12:27:51.553578 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.553550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" event={"ID":"55b5d07ef5da4396d8fc84d8570a1cf0","Type":"ContainerStarted","Data":"565cfb9fb82bc90ca65f63d5c277719a5a045096d79e53911ba4ba0efca9ba91"} Aug 04 12:27:51.587037 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.582176 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 12:27:51.588309 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5xd\" (UniqueName: \"kubernetes.io/projected/4fc0096c-c114-448c-ac13-639b117d40f3-kube-api-access-rp5xd\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:51.588309 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-cni-binary-copy\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.588309 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588136 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-ovn\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588309 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-log-socket\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588309 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-dbus\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-slash\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-etc-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-run-ovn-kubernetes\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-cni-bin\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-ovnkube-script-lib\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.588959 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.588959 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-cnibin\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.588959 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.588959 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255tp\" (UniqueName: \"kubernetes.io/projected/72ca687e-554b-41b0-9feb-40167b647220-kube-api-access-255tp\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589171 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.588975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-device-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.589171 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.589275 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-system-cni-dir\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.589275 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-cni-netd\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589275 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:51.589275 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc7df39c-e1e8-45a5-a852-a75592bd193e-host\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.589463 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-env-overrides\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589463 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc7df39c-e1e8-45a5-a852-a75592bd193e-serviceca\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.589463 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-kubelet-config\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.589611 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.589611 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.589611 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:51.589611 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-node-log\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589804 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-var-lib-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589804 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-systemd\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589804 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-run-netns\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.589804 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d82696e-2556-495c-93d3-4b882156c2a4-konnectivity-ca\") pod \"konnectivity-agent-8j5qs\" (UID: \"7d82696e-2556-495c-93d3-4b882156c2a4\") " pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.589804 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.589804 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn6d8\" (UniqueName: \"kubernetes.io/projected/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-kube-api-access-mn6d8\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.590071 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-os-release\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.590071 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.589853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnvd\" (UniqueName: \"kubernetes.io/projected/564ce3c2-fbd1-400a-993d-7310abe7a910-kube-api-access-8xnvd\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.590071 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.590021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.590071 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.590051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-ovnkube-config\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.590290 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.590084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ljn\" (UniqueName: \"kubernetes.io/projected/fc7df39c-e1e8-45a5-a852-a75592bd193e-kube-api-access-q4ljn\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.593223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.592189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-systemd-units\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.593223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.592249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.593223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.592291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.593223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.592322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-kubelet\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.593223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.592356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ca687e-554b-41b0-9feb-40167b647220-ovn-node-metrics-cert\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.593223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.592431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d82696e-2556-495c-93d3-4b882156c2a4-agent-certs\") pod \"konnectivity-agent-8j5qs\" (UID: \"7d82696e-2556-495c-93d3-4b882156c2a4\") " pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.594139 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-system-cni-dir\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-cni-binary-copy\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-ovn\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-log-socket\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-sys-fs\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-dbus\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-slash\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595584 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-etc-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-run-ovn-kubernetes\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.595690 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-cni-bin\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.596333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-ovnkube-script-lib\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.596425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.596503 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-registration-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.596562 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-cnibin\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.596622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.596952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-device-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.597521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.597075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-etc-selinux\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.598420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.597618 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fc7df39c-e1e8-45a5-a852-a75592bd193e-host\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.598420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.597704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-node-log\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.598420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.597791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-kubelet-config\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:51.598420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.598083 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-cni-netd\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.598420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.598253 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fc7df39c-e1e8-45a5-a852-a75592bd193e-serviceca\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.598420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.598305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-socket-dir\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.598714 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.598612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-env-overrides\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.598714 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.598667 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:51.598804 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.598754 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:52.098733317 +0000 UTC m=+3.054329114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:51.598967 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.598941 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.599049 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-systemd-units\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.599289 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.599450 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599326 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-kubelet\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.599538 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-var-lib-openvswitch\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.599538 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599515 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/564ce3c2-fbd1-400a-993d-7310abe7a910-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.599647 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.599574 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:51.599722 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.599698 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:52.099641767 +0000 UTC m=+3.055237551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:51.599790 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-run-systemd\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.599937 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.599909 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-run-netns\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.600026 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.600000 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/564ce3c2-fbd1-400a-993d-7310abe7a910-os-release\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.600434 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.600411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ca687e-554b-41b0-9feb-40167b647220-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.600517 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.600460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d82696e-2556-495c-93d3-4b882156c2a4-konnectivity-ca\") pod \"konnectivity-agent-8j5qs\" (UID: \"7d82696e-2556-495c-93d3-4b882156c2a4\") " pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.601507 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.601461 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72ca687e-554b-41b0-9feb-40167b647220-ovnkube-config\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.601600 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.601564 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d82696e-2556-495c-93d3-4b882156c2a4-agent-certs\") pod \"konnectivity-agent-8j5qs\" (UID: \"7d82696e-2556-495c-93d3-4b882156c2a4\") " pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.605404 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.605318 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:27:51.605404 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.605352 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:27:51.605404 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.605366 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:51.605588 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:51.605440 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:52.105420064 +0000 UTC m=+3.061015847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:51.605588 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.605439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72ca687e-554b-41b0-9feb-40167b647220-ovn-node-metrics-cert\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.607290 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.606923 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5xd\" (UniqueName: \"kubernetes.io/projected/4fc0096c-c114-448c-ac13-639b117d40f3-kube-api-access-rp5xd\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:51.614654 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.614625 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-255tp\" (UniqueName: \"kubernetes.io/projected/72ca687e-554b-41b0-9feb-40167b647220-kube-api-access-255tp\") pod \"ovnkube-node-qch7k\" (UID: \"72ca687e-554b-41b0-9feb-40167b647220\") " pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.615738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.615710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnvd\" (UniqueName: \"kubernetes.io/projected/564ce3c2-fbd1-400a-993d-7310abe7a910-kube-api-access-8xnvd\") pod \"multus-additional-cni-plugins-tlnnv\" (UID: \"564ce3c2-fbd1-400a-993d-7310abe7a910\") " pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:51.615849 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.615714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ljn\" (UniqueName: \"kubernetes.io/projected/fc7df39c-e1e8-45a5-a852-a75592bd193e-kube-api-access-q4ljn\") pod \"node-ca-9dwhz\" (UID: \"fc7df39c-e1e8-45a5-a852-a75592bd193e\") " pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.617044 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.617023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn6d8\" (UniqueName: \"kubernetes.io/projected/dc600fb5-05bf-4cfb-81ba-a4308534c2b3-kube-api-access-mn6d8\") pod \"aws-ebs-csi-driver-node-7rtlp\" (UID: \"dc600fb5-05bf-4cfb-81ba-a4308534c2b3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.679376 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.679342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bc27s" Aug 04 12:27:51.688374 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.688335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qzvv" Aug 04 12:27:51.696217 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.696189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jqglq" Aug 04 12:27:51.701570 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.701538 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7vggh" Aug 04 12:27:51.708222 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.708197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9dwhz" Aug 04 12:27:51.719487 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.719248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:27:51.726245 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.726220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:27:51.732155 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.732114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" Aug 04 12:27:51.737254 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:51.737063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" Aug 04 12:27:52.195891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:52.195848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:52.196093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:52.195901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:52.196093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:52.195943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:52.196093 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196027 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:52.196093 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196056 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:52.196093 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196089 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:27:52.196326 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196103 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:53.196082248 +0000 UTC m=+4.151678020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:52.196326 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196110 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:27:52.196326 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196138 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:52.196326 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196139 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:53.196113822 +0000 UTC m=+4.151709595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:52.196326 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:52.196198 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:53.196180396 +0000 UTC m=+4.151776169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:52.420159 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:52.420104 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 12:22:51 +0000 UTC" deadline="2028-04-29 13:53:06.4190964 +0000 UTC" Aug 04 12:27:52.420159 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:52.420154 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15217h25m13.998945415s" Aug 04 12:27:52.711174 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:52.711137 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc600fb5_05bf_4cfb_81ba_a4308534c2b3.slice/crio-e7c236cee5d9e0c574af71eda396d8bd1be7d6c2ad478243c9df3934b8ce7f19 WatchSource:0}: Error finding container e7c236cee5d9e0c574af71eda396d8bd1be7d6c2ad478243c9df3934b8ce7f19: Status 404 returned error can't find the container with id e7c236cee5d9e0c574af71eda396d8bd1be7d6c2ad478243c9df3934b8ce7f19 Aug 04 12:27:52.713599 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:52.713428 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7df39c_e1e8_45a5_a852_a75592bd193e.slice/crio-20434a45ae11f0be892a07fd71431e2504b717fe1d3b9fc88aed29f05495062f WatchSource:0}: Error finding container 20434a45ae11f0be892a07fd71431e2504b717fe1d3b9fc88aed29f05495062f: Status 404 returned error can't find the container with id 20434a45ae11f0be892a07fd71431e2504b717fe1d3b9fc88aed29f05495062f Aug 04 12:27:52.714577 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:52.714557 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564ce3c2_fbd1_400a_993d_7310abe7a910.slice/crio-dfe2b47093119143cb07fa245f5b3f9ce05829780ffc9e57d26c21f3fc78dee5 WatchSource:0}: Error finding container dfe2b47093119143cb07fa245f5b3f9ce05829780ffc9e57d26c21f3fc78dee5: Status 404 returned error can't find the container with id dfe2b47093119143cb07fa245f5b3f9ce05829780ffc9e57d26c21f3fc78dee5 Aug 04 12:27:52.717424 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:52.716696 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82696e_2556_495c_93d3_4b882156c2a4.slice/crio-a36b1a0f020fa2dc86a4f61f2b6fadeecf23217c53f67bc879f7787806e79698 WatchSource:0}: Error finding container a36b1a0f020fa2dc86a4f61f2b6fadeecf23217c53f67bc879f7787806e79698: Status 404 returned error can't find the container with id a36b1a0f020fa2dc86a4f61f2b6fadeecf23217c53f67bc879f7787806e79698 Aug 04 12:27:52.719391 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:52.718593 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc89e5a3_43c5_4413_85ce_67c1df2eb916.slice/crio-f0776c129feda2db9f6812e221ec9fc0deb10102a40c4721d060d3cfc4d5ffb4 WatchSource:0}: Error finding container f0776c129feda2db9f6812e221ec9fc0deb10102a40c4721d060d3cfc4d5ffb4: Status 404 returned error can't find the container with id f0776c129feda2db9f6812e221ec9fc0deb10102a40c4721d060d3cfc4d5ffb4 Aug 04 12:27:52.719391 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:27:52.718757 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3b799a_900a_4bff_be86_986adc410241.slice/crio-1e017a01c1e0c6d12d0c736415ada1c1969713f9cd4fe7e396b58357568fe00d WatchSource:0}: Error finding container 1e017a01c1e0c6d12d0c736415ada1c1969713f9cd4fe7e396b58357568fe00d: Status 404 returned error can't find the container with id 1e017a01c1e0c6d12d0c736415ada1c1969713f9cd4fe7e396b58357568fe00d Aug 04 12:27:53.201224 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.200903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:53.201224 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.201201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:53.201402 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.201243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:53.201402 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201100 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:27:53.201402 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201353 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:27:53.201402 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201363 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:53.201602 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201407 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:55.201394011 +0000 UTC m=+6.156989782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:53.201711 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201680 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:53.201841 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201721 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:55.20170878 +0000 UTC m=+6.157304552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:53.201841 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201322 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:53.201841 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.201765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:55.201745867 +0000 UTC m=+6.157341639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:53.420975 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.420867 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 12:22:51 +0000 UTC" deadline="2028-05-03 00:57:39.63827791 +0000 UTC" Aug 04 12:27:53.420975 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.420907 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15300h29m46.217375557s" Aug 04 12:27:53.548833 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.547919 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:53.548833 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.548052 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:27:53.548833 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.548494 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:53.548833 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.548599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:27:53.548833 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.548674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:53.548833 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:53.548745 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:27:53.570559 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.570440 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" event={"ID":"1a2944b9797870016f92974beffaabc3","Type":"ContainerStarted","Data":"72d96ea1692cafe9c89478622ee3c82a70aea92ddfeced99477716ea2b167ea9"} Aug 04 12:27:53.586907 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.586857 2578 generic.go:356] "Generic (PLEG): container finished" podID="55b5d07ef5da4396d8fc84d8570a1cf0" containerID="e5d7f1f90ec15f814253dd734c6906a53279004246b8835552f6e800d48b6669" exitCode=0 Aug 04 12:27:53.587062 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.586950 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" event={"ID":"55b5d07ef5da4396d8fc84d8570a1cf0","Type":"ContainerDied","Data":"e5d7f1f90ec15f814253dd734c6906a53279004246b8835552f6e800d48b6669"} Aug 04 12:27:53.589574 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.588675 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.ec2.internal" podStartSLOduration=3.588656994 podStartE2EDuration="3.588656994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:27:53.587803879 +0000 UTC m=+4.543399677" watchObservedRunningTime="2026-08-04 12:27:53.588656994 +0000 UTC m=+4.544252807" Aug 04 12:27:53.603838 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.603799 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qzvv" event={"ID":"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea","Type":"ContainerStarted","Data":"955056bfe9433d9231a858f5f7d8f2f12871d4afb0482d6eea816df53c27cff5"} Aug 04 12:27:53.606976 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.606943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"4479b0f32a10b607b17368e7461970177441381134d2456397a75e3f122d94b1"} Aug 04 12:27:53.629268 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.629230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jqglq" event={"ID":"b5460bc2-716e-485b-a943-b65d58b0bb32","Type":"ContainerStarted","Data":"4d3f58c8f1d3e0c1ccf69a0e974c49ba9019b13dffc8a22a3b3ddbe73ac6ef79"} Aug 04 12:27:53.635862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.635827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bc27s" event={"ID":"cc89e5a3-43c5-4413-85ce-67c1df2eb916","Type":"ContainerStarted","Data":"f0776c129feda2db9f6812e221ec9fc0deb10102a40c4721d060d3cfc4d5ffb4"} Aug 04 12:27:53.645372 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.645324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8j5qs" event={"ID":"7d82696e-2556-495c-93d3-4b882156c2a4","Type":"ContainerStarted","Data":"a36b1a0f020fa2dc86a4f61f2b6fadeecf23217c53f67bc879f7787806e79698"} Aug 04 12:27:53.654809 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.654723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" event={"ID":"dc600fb5-05bf-4cfb-81ba-a4308534c2b3","Type":"ContainerStarted","Data":"e7c236cee5d9e0c574af71eda396d8bd1be7d6c2ad478243c9df3934b8ce7f19"} Aug 04 12:27:53.663846 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.663769 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7vggh" event={"ID":"8f3b799a-900a-4bff-be86-986adc410241","Type":"ContainerStarted","Data":"1e017a01c1e0c6d12d0c736415ada1c1969713f9cd4fe7e396b58357568fe00d"} Aug 04 12:27:53.677386 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.677340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerStarted","Data":"dfe2b47093119143cb07fa245f5b3f9ce05829780ffc9e57d26c21f3fc78dee5"} Aug 04 12:27:53.684910 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:53.684818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9dwhz" event={"ID":"fc7df39c-e1e8-45a5-a852-a75592bd193e","Type":"ContainerStarted","Data":"20434a45ae11f0be892a07fd71431e2504b717fe1d3b9fc88aed29f05495062f"} Aug 04 12:27:54.704732 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:54.704693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" event={"ID":"55b5d07ef5da4396d8fc84d8570a1cf0","Type":"ContainerStarted","Data":"2f276f5bb8e9fa890f22de336bcc403a774cad1854e5685bde771bf73ce87722"} Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:55.222560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:55.222608 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:55.222651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222727 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222770 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222776 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222793 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:59.222774162 +0000 UTC m=+10.178369940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222797 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222811 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222820 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:59.222803902 +0000 UTC m=+10.178399676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:55.222851 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.222845 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:27:59.22283558 +0000 UTC m=+10.178431355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:55.548323 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:55.548244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:55.548473 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.548386 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:27:55.548473 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:55.548244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:55.548591 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.548511 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:27:55.548591 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:55.548586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:55.548697 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:55.548653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:27:57.549275 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:57.549245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:57.549704 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:57.549379 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:27:57.549704 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:57.549245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:57.549704 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:57.549501 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:27:57.549704 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:57.549563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:57.549704 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:57.549654 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:27:59.257795 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:59.257751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:59.258279 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:59.257839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:59.258279 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:59.257878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:59.258279 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258006 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:59.258279 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258079 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:07.25805579 +0000 UTC m=+18.213651586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:27:59.258507 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258463 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:59.258553 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258523 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:07.258505742 +0000 UTC m=+18.214101518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:27:59.258611 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258598 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:27:59.258642 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258620 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:27:59.258642 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258632 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:59.258950 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.258657 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:07.258649582 +0000 UTC m=+18.214245354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:27:59.548104 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:59.548019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:27:59.548271 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:59.548134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:27:59.548271 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.548168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:27:59.548271 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.548205 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:27:59.548271 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:27:59.548249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:27:59.548501 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:27:59.548317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:01.547234 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:01.547197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:01.547674 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:01.547343 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:01.547674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:01.547197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:01.547674 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:01.547444 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:01.547674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:01.547197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:01.547674 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:01.547519 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:03.550755 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:03.550673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:03.550755 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:03.550715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:03.551200 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:03.550774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:03.551200 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:03.550788 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:03.551200 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:03.550865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:03.551200 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:03.550945 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:05.547812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:05.547777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:05.548259 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:05.547905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:05.548259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:05.547777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:05.548259 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:05.548000 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:05.548259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:05.547777 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:05.548259 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:05.548063 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:07.324007 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:07.323972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:07.324021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:07.324052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324169 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324172 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324219 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:23.32420508 +0000 UTC m=+34.279800851 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324249 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:23.324229916 +0000 UTC m=+34.279825708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324181 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324282 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324296 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:28:07.324510 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.324343 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:23.324331809 +0000 UTC m=+34.279927582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:28:07.547640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:07.547551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:07.547640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:07.547580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:07.547864 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.547693 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:07.547864 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.547787 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:07.547864 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:07.547832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:07.548001 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:07.547895 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:09.548401 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:09.548365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:09.548781 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:09.548481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:09.548781 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:09.548494 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:09.548781 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:09.548601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:09.548781 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:09.548606 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:09.548781 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:09.548655 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:11.551305 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.549653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:11.551305 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:11.549784 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:11.551305 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.550723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:11.551305 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:11.550825 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:11.551305 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.550874 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:11.551305 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:11.550931 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:11.740602 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.740448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"6d2fe81e76388f33b381c4f2b6745fba52aef9c0d78881d195f9ff5b8e8b5f35"} Aug 04 12:28:11.742144 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.742011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jqglq" event={"ID":"b5460bc2-716e-485b-a943-b65d58b0bb32","Type":"ContainerStarted","Data":"2a5f01b88aa2ed962e07c175af61753600202b03ad00600c2199e8fcb7e00096"} Aug 04 12:28:11.744033 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.744005 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bc27s" event={"ID":"cc89e5a3-43c5-4413-85ce-67c1df2eb916","Type":"ContainerStarted","Data":"3867ea06c6ee1d7043cdea18f1b21e0afcb61d3adeb43889a440861ab9fb09cd"} Aug 04 12:28:11.745688 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.745360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8j5qs" event={"ID":"7d82696e-2556-495c-93d3-4b882156c2a4","Type":"ContainerStarted","Data":"dbb43b1c11ab0f10e4bbbf7589d96ad4d83624a32acf94fe7038492e0efc049f"} Aug 04 12:28:11.747797 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.747767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" event={"ID":"dc600fb5-05bf-4cfb-81ba-a4308534c2b3","Type":"ContainerStarted","Data":"6f62fb3f7f7496e9ec8c929679a1affd83c27545c7e3527a1d7c5a3ff16100e9"} Aug 04 12:28:11.749112 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.749073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7vggh" event={"ID":"8f3b799a-900a-4bff-be86-986adc410241","Type":"ContainerStarted","Data":"ff54aea50f2080aae8fa5fe986f712c2581028b992d11a8355e26eccd457d00a"} Aug 04 12:28:11.750942 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.750703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerStarted","Data":"09ddbdbc61db85211a4d5e001b2d170e1466f29046f3c9eb3f7825d7ae28c5f7"} Aug 04 12:28:11.752225 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.752184 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9dwhz" event={"ID":"fc7df39c-e1e8-45a5-a852-a75592bd193e","Type":"ContainerStarted","Data":"9dcb244a5c8ca9e1983b39a2d95ab863bfbc2396648c1a77a087198047873912"} Aug 04 12:28:11.764301 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.764266 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jqglq" podStartSLOduration=4.004466789 podStartE2EDuration="22.764253601s" totalImagesPullingTime="18.759786812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:11.763769647 +0000 UTC m=+22.719365440" watchObservedRunningTime="2026-08-04 12:28:11.764253601 +0000 UTC m=+22.719849396" Aug 04 12:28:11.764423 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.764407 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.ec2.internal" podStartSLOduration=21.764402987 podStartE2EDuration="21.764402987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:27:54.723787835 +0000 UTC m=+5.679383631" watchObservedRunningTime="2026-08-04 12:28:11.764402987 +0000 UTC m=+22.719998781" Aug 04 12:28:11.810781 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.810735 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8j5qs" podStartSLOduration=8.325642829 podStartE2EDuration="22.810719209s" totalImagesPullingTime="14.48507638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:11.786065261 +0000 UTC m=+22.741661054" watchObservedRunningTime="2026-08-04 12:28:11.810719209 +0000 UTC m=+22.766315003" Aug 04 12:28:11.827311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.827268 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7vggh" podStartSLOduration=4.062296441 podStartE2EDuration="22.827252895s" totalImagesPullingTime="18.764956454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:11.826953966 +0000 UTC m=+22.782549761" watchObservedRunningTime="2026-08-04 12:28:11.827252895 +0000 UTC m=+22.782848690" Aug 04 12:28:11.847058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:11.847000 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9dwhz" podStartSLOduration=4.078572661 podStartE2EDuration="22.846977984s" totalImagesPullingTime="18.768405323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:11.846813515 +0000 UTC m=+22.802409357" watchObservedRunningTime="2026-08-04 12:28:11.846977984 +0000 UTC m=+22.802573779" Aug 04 12:28:12.364651 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.364409 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:12.465343 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.465304 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:12.629164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.629110 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:12.756616 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.756591 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:28:12.756922 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.756901 2578 generic.go:356] "Generic (PLEG): container finished" podID="72ca687e-554b-41b0-9feb-40167b647220" containerID="0e8c24eb8a54adccd1ba2421a46d2c72604da67aab7c62a291a3cacee1c2f3b0" exitCode=1 Aug 04 12:28:12.756993 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.756971 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"abe5125c66a0f74d43d7d4e3734350393eecf3051809acff54ec6c9ef6bdd96a"} Aug 04 12:28:12.757137 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.757100 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"e5c993914062b8612d2ab9b9135cc8d714895400734ed87979494925f0959435"} Aug 04 12:28:12.757200 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.757167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"1c675dac92983727cc59d119db2adff67613f4acb6871bab7c70f6649a6dd4aa"} Aug 04 12:28:12.757240 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.757199 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"621724cde7135f2d0a077a8ba8d56834df8ed64da7d87a14628bb7a3b7d8ac26"} Aug 04 12:28:12.757240 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.757230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerDied","Data":"0e8c24eb8a54adccd1ba2421a46d2c72604da67aab7c62a291a3cacee1c2f3b0"} Aug 04 12:28:12.758413 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.758393 2578 generic.go:356] "Generic (PLEG): container finished" podID="564ce3c2-fbd1-400a-993d-7310abe7a910" containerID="09ddbdbc61db85211a4d5e001b2d170e1466f29046f3c9eb3f7825d7ae28c5f7" exitCode=0 Aug 04 12:28:12.758526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.758492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerDied","Data":"09ddbdbc61db85211a4d5e001b2d170e1466f29046f3c9eb3f7825d7ae28c5f7"} Aug 04 12:28:12.778790 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:12.778743 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bc27s" podStartSLOduration=4.939040167 podStartE2EDuration="23.778729156s" totalImagesPullingTime="18.839688989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:12.778626144 +0000 UTC m=+23.734221937" watchObservedRunningTime="2026-08-04 12:28:12.778729156 +0000 UTC m=+23.734324950" Aug 04 12:28:13.251688 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.251664 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 12:28:13.442608 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.442506 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T12:28:13.251682869Z","UUID":"84c926ac-487c-4b2e-89b4-71ad81553074","Handler":null,"Name":"","Endpoint":""} Aug 04 12:28:13.444075 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.444054 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 12:28:13.444187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.444090 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 12:28:13.547355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.547320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:13.547514 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.547321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:13.547514 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:13.547455 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:13.547657 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.547321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:13.547657 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:13.547543 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:13.547657 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:13.547638 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:13.763255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.763181 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" event={"ID":"dc600fb5-05bf-4cfb-81ba-a4308534c2b3","Type":"ContainerStarted","Data":"f15bd38a62ab246e83b999c2580bf607d8631ef550e1f8beef3168d8cce142da"} Aug 04 12:28:13.764821 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.764779 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qzvv" event={"ID":"7ee5740e-b7c7-4975-a8b4-2eb2ae6792ea","Type":"ContainerStarted","Data":"973439cf1cb743ff4b61908bcb8911bc03386fa2883b5264eb4d04c44f783dfe"} Aug 04 12:28:13.782134 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.782067 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7qzvv" podStartSLOduration=6.021920739 podStartE2EDuration="24.782051843s" totalImagesPullingTime="18.760131104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:13.781587684 +0000 UTC m=+24.737183475" watchObservedRunningTime="2026-08-04 12:28:13.782051843 +0000 UTC m=+24.737647637" Aug 04 12:28:13.861092 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:13.861038 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.152973 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.152897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.250951 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.250912 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.437668 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.437568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.545160 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.545115 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.643691 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.643505 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.706677 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.706587 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:28:14.707507 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.707460 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:28:14.732724 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.732695 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:14.771294 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.771266 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:28:14.771784 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.771751 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"844842c663e0e4d73fc22ca60a72373f3ff28beb4cf4aed88409d84223f6eec2"} Aug 04 12:28:14.774185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.774156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" event={"ID":"dc600fb5-05bf-4cfb-81ba-a4308534c2b3","Type":"ContainerStarted","Data":"64ccead5f85a0a4f44446b0c9451f9714e445b3324fe6667ae4cdf53238fc954"} Aug 04 12:28:14.810940 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.810890 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rtlp" podStartSLOduration=4.42533769 podStartE2EDuration="25.810872952s" totalImagesPullingTime="21.385535262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:14.810629637 +0000 UTC m=+25.766225431" watchObservedRunningTime="2026-08-04 12:28:14.810872952 +0000 UTC m=+25.766468743" Aug 04 12:28:14.964191 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:14.964081 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:15.547245 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:15.547212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:15.547456 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:15.547247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:15.547456 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:15.547369 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:15.547586 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:15.547469 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:15.547741 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:15.547725 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:15.547838 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:15.547820 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:15.629321 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:15.629286 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:28:15.630110 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:15.630024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8j5qs" Aug 04 12:28:15.942906 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:15.942879 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:16.351672 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:16.351645 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:16.559347 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:16.559309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:16.752823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:16.752739 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:17.150680 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.150642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:17.547932 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.547898 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:17.547932 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.547921 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:17.548107 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:17.548024 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:17.548107 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.548071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:17.548239 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:17.548180 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:17.548292 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:17.548281 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:17.554249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.554228 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:17.655843 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.655690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:17.756677 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.756444 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:17.783568 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.783455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerStarted","Data":"51daa53bc9e260ad42c9c1e94b394ba5468ea2ffc71ccdf8bc5039936d1dfce8"} Aug 04 12:28:17.787092 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.787068 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:28:17.787545 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.787502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"c22547d2079f8e4ab5426f4d6189535bb973009a606ae9cc1ccd347d77b6209e"} Aug 04 12:28:17.787699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.787682 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:28:17.787764 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.787755 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:28:17.787915 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.787900 2578 scope.go:117] "RemoveContainer" containerID="0e8c24eb8a54adccd1ba2421a46d2c72604da67aab7c62a291a3cacee1c2f3b0" Aug 04 12:28:17.805004 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:17.804986 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:28:18.273190 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.273137 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:18.791273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.791187 2578 generic.go:356] "Generic (PLEG): container finished" podID="564ce3c2-fbd1-400a-993d-7310abe7a910" containerID="51daa53bc9e260ad42c9c1e94b394ba5468ea2ffc71ccdf8bc5039936d1dfce8" exitCode=0 Aug 04 12:28:18.791273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.791249 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerDied","Data":"51daa53bc9e260ad42c9c1e94b394ba5468ea2ffc71ccdf8bc5039936d1dfce8"} Aug 04 12:28:18.795290 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.795271 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:28:18.795719 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.795690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" event={"ID":"72ca687e-554b-41b0-9feb-40167b647220","Type":"ContainerStarted","Data":"d6596fdf48a5d4afa42f289ae9393200aa5cf95d0782b8cb340e54d0f852cabc"} Aug 04 12:28:18.795991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.795976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:28:18.812009 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.811979 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:28:18.856694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.856639 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" podStartSLOduration=11.049731854 podStartE2EDuration="29.856624008s" totalImagesPullingTime="18.806892154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:18.856254019 +0000 UTC m=+29.811849812" watchObservedRunningTime="2026-08-04 12:28:18.856624008 +0000 UTC m=+29.812219803" Aug 04 12:28:18.979372 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:18.979341 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:19.168366 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.168330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:19.254041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.253995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:19.351777 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.351745 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:19.548280 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.548252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:19.548424 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.548321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:19.548424 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:19.548339 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:19.548500 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:19.548420 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:19.548500 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.548456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:19.548562 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:19.548534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:19.558728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.558699 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:19.759484 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.759404 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:19.800225 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.800190 2578 generic.go:356] "Generic (PLEG): container finished" podID="564ce3c2-fbd1-400a-993d-7310abe7a910" containerID="4d374764583d673201e02108bcd5a73df9d406b407317c4a83cd8a862273e3e8" exitCode=0 Aug 04 12:28:19.800371 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:19.800228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerDied","Data":"4d374764583d673201e02108bcd5a73df9d406b407317c4a83cd8a862273e3e8"} Aug 04 12:28:20.752738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:20.752659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:20.804524 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:20.804488 2578 generic.go:356] "Generic (PLEG): container finished" podID="564ce3c2-fbd1-400a-993d-7310abe7a910" containerID="7f7d7fd5a1a3225794cbfe585c11cb0f7ff408c9ac765c13332fced7249066be" exitCode=0 Aug 04 12:28:20.804672 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:20.804586 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerDied","Data":"7f7d7fd5a1a3225794cbfe585c11cb0f7ff408c9ac765c13332fced7249066be"} Aug 04 12:28:20.946326 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:20.946286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:21.345684 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:21.345652 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:21.466891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:21.466850 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:21.546885 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:21.546808 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:21.549999 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:21.549974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:21.550165 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:21.550095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:21.550165 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:21.550101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:21.550165 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:21.550138 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:21.550317 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:21.550243 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:21.550362 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:21.550342 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:22.148406 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:22.148370 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:22.250313 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:22.250280 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:22.546949 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:22.546852 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:22.957616 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:22.957568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:23.152278 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.152230 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:23.352320 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.352288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:23.352495 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.352329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:23.352495 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.352376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:23.352495 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352446 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:28:23.352495 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352452 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 12:28:23.352698 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352515 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs podName:4fc0096c-c114-448c-ac13-639b117d40f3 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:55.352495044 +0000 UTC m=+66.308090820 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs") pod "network-metrics-daemon-z28dh" (UID: "4fc0096c-c114-448c-ac13-639b117d40f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 12:28:23.352698 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352529 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 12:28:23.352698 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352558 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 12:28:23.352698 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352571 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwlw9 for pod openshift-network-diagnostics/network-check-target-46dcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:28:23.352698 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret podName:c2f3b6b1-86db-4fb6-9afd-e305159d16a2 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:55.35252565 +0000 UTC m=+66.308121422 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret") pod "global-pull-secret-syncer-kd7rr" (UID: "c2f3b6b1-86db-4fb6-9afd-e305159d16a2") : object "kube-system"/"original-pull-secret" not registered Aug 04 12:28:23.352698 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.352632 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9 podName:e7a58fbf-6eff-4c99-8373-7cc20e14b667 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:55.352616448 +0000 UTC m=+66.308212236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwlw9" (UniqueName: "kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9") pod "network-check-target-46dcw" (UID: "e7a58fbf-6eff-4c99-8373-7cc20e14b667") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 12:28:23.547081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.547038 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:23.547081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.547079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:23.547081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.547090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:23.547390 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.547242 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:23.547390 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.547375 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:23.547498 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:23.547457 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:23.757110 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:23.756838 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:24.152602 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:24.152559 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:24.356785 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:24.356755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:24.565041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:24.564961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:25.547027 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:25.546933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:25.547027 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:25.546967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:25.547583 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:25.546933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:25.547583 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:25.547057 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:25.547583 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:25.547152 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:25.547583 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:25.547202 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:25.819376 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:25.819291 2578 generic.go:356] "Generic (PLEG): container finished" podID="564ce3c2-fbd1-400a-993d-7310abe7a910" containerID="dda38394adbc47bfba1bd26efdebc85e19fda3e02d40b1a2dfd8a82820e3b24b" exitCode=0 Aug 04 12:28:25.819376 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:25.819350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerDied","Data":"dda38394adbc47bfba1bd26efdebc85e19fda3e02d40b1a2dfd8a82820e3b24b"} Aug 04 12:28:26.827687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:26.827652 2578 generic.go:356] "Generic (PLEG): container finished" podID="564ce3c2-fbd1-400a-993d-7310abe7a910" containerID="3f0fa1f62d1fb1faf433396dc016aa1bbf5e8a450e1e67f49d80530604d98615" exitCode=0 Aug 04 12:28:26.828046 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:26.827700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerDied","Data":"3f0fa1f62d1fb1faf433396dc016aa1bbf5e8a450e1e67f49d80530604d98615"} Aug 04 12:28:27.547756 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:27.547715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:27.547756 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:27.547746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:27.547964 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:27.547715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:27.547964 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:27.547836 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:27.547964 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:27.547901 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:27.548085 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:27.547988 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:27.832968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:27.832887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" event={"ID":"564ce3c2-fbd1-400a-993d-7310abe7a910","Type":"ContainerStarted","Data":"20198acdb0b4039d73b1a6d053a9026d2038ab66ac7aa60801e41cc5c0df8778"} Aug 04 12:28:27.863154 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:27.863073 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tlnnv" podStartSLOduration=9.661764486 podStartE2EDuration="38.863059755s" totalImagesPullingTime="29.201295269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:27.862498019 +0000 UTC m=+38.818093813" watchObservedRunningTime="2026-08-04 12:28:27.863059755 +0000 UTC m=+38.818655551" Aug 04 12:28:28.643809 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:28.643773 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:28.718187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:28.718155 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:28.785002 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:28.784968 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:28.928295 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:28.928220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:29.301577 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.301492 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:29.548898 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.548865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:29.549078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.548947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:29.549078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.548983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:29.549078 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:29.549001 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:29.549078 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:29.549046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:29.549278 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:29.549104 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:29.593930 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.593901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:29.714288 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.714235 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:29.811326 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.811288 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:29.908144 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:29.908055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:30.109687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.109654 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:30.209312 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.209220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:30.326728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.326702 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:30.520037 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.519959 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:30.910929 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.910834 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd7rr"] Aug 04 12:28:30.911099 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.910952 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:30.911099 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:30.911058 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:30.919692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.918423 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z28dh"] Aug 04 12:28:30.919692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.918527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-46dcw"] Aug 04 12:28:30.919692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.918614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:30.919692 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:30.918728 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:30.919692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:30.918994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:30.919692 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:30.919151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:31.052492 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:31.052467 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:31.544093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:31.544058 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:31.614046 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:31.614012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:32.547712 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:32.547684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:32.548057 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:32.547691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:32.548057 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:32.547795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:32.548057 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:32.547799 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd7rr" podUID="c2f3b6b1-86db-4fb6-9afd-e305159d16a2" Aug 04 12:28:32.548057 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:32.547871 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z28dh" podUID="4fc0096c-c114-448c-ac13-639b117d40f3" Aug 04 12:28:32.548057 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:32.547942 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46dcw" podUID="e7a58fbf-6eff-4c99-8373-7cc20e14b667" Aug 04 12:28:33.176024 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:33.175993 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:33.273516 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:33.273484 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:33.365219 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:33.365189 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:33.640378 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:33.640347 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:33.904386 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:33.904295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:34.008837 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.008803 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:34.390028 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.389998 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-46.ec2.internal" event="NodeReady" Aug 04 12:28:34.390236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.390151 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 12:28:34.452564 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.452533 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gwf96"] Aug 04 12:28:34.473817 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.473793 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ncsgj"] Aug 04 12:28:34.473947 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.473914 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.477221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.477021 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 12:28:34.477337 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.477022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 12:28:34.477337 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.477022 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p4v7x\"" Aug 04 12:28:34.493686 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.493662 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gwf96"] Aug 04 12:28:34.493795 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.493724 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ncsgj"] Aug 04 12:28:34.493795 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.493742 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ndt2d"] Aug 04 12:28:34.493795 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.493671 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.497796 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.497771 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 12:28:34.497910 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.497892 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 12:28:34.497979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.497909 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-25txl\"" Aug 04 12:28:34.497979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.497939 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 12:28:34.508708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.508689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ndt2d"] Aug 04 12:28:34.508792 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.508786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.511599 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.511579 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 12:28:34.511706 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.511609 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4bx8n\"" Aug 04 12:28:34.511706 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.511665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 12:28:34.511934 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.511919 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 12:28:34.511984 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.511953 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 12:28:34.547794 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.547764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:34.547956 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.547768 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:34.548013 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.547769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:34.550587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.550569 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 12:28:34.550725 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.550659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p2dbw\"" Aug 04 12:28:34.550992 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.550976 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st96r\"" Aug 04 12:28:34.551087 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.551034 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 12:28:34.551087 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.551034 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 12:28:34.551191 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.551037 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 12:28:34.624411 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.624379 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:34.636446 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-config-volume\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.636552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed9d7305-2faa-4e74-a7b6-f90c95f59092-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.636552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/546cdf59-c034-4ab6-a386-f8ddfe14aeb3-cert\") pod \"ingress-canary-ncsgj\" (UID: \"546cdf59-c034-4ab6-a386-f8ddfe14aeb3\") " pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.636552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694hc\" (UniqueName: \"kubernetes.io/projected/ed9d7305-2faa-4e74-a7b6-f90c95f59092-kube-api-access-694hc\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.636552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-tmp-dir\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.636552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8hv\" (UniqueName: \"kubernetes.io/projected/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-kube-api-access-6v8hv\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.636751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7vz\" (UniqueName: \"kubernetes.io/projected/546cdf59-c034-4ab6-a386-f8ddfe14aeb3-kube-api-access-mj7vz\") pod \"ingress-canary-ncsgj\" (UID: \"546cdf59-c034-4ab6-a386-f8ddfe14aeb3\") " pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.636751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed9d7305-2faa-4e74-a7b6-f90c95f59092-crio-socket\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.636751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed9d7305-2faa-4e74-a7b6-f90c95f59092-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.636751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-metrics-tls\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.636751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.636705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed9d7305-2faa-4e74-a7b6-f90c95f59092-data-volume\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.690753 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.690676 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:34.737516 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed9d7305-2faa-4e74-a7b6-f90c95f59092-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.737516 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-metrics-tls\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.737728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed9d7305-2faa-4e74-a7b6-f90c95f59092-data-volume\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.737728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-config-volume\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.737728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed9d7305-2faa-4e74-a7b6-f90c95f59092-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.737728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/546cdf59-c034-4ab6-a386-f8ddfe14aeb3-cert\") pod \"ingress-canary-ncsgj\" (UID: \"546cdf59-c034-4ab6-a386-f8ddfe14aeb3\") " pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.737728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694hc\" (UniqueName: \"kubernetes.io/projected/ed9d7305-2faa-4e74-a7b6-f90c95f59092-kube-api-access-694hc\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.737728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-tmp-dir\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.738010 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8hv\" (UniqueName: \"kubernetes.io/projected/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-kube-api-access-6v8hv\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.738010 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7vz\" (UniqueName: \"kubernetes.io/projected/546cdf59-c034-4ab6-a386-f8ddfe14aeb3-kube-api-access-mj7vz\") pod \"ingress-canary-ncsgj\" (UID: \"546cdf59-c034-4ab6-a386-f8ddfe14aeb3\") " pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.738010 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed9d7305-2faa-4e74-a7b6-f90c95f59092-crio-socket\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.738010 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.737977 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed9d7305-2faa-4e74-a7b6-f90c95f59092-crio-socket\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.738347 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.738264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-tmp-dir\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.738347 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.738274 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed9d7305-2faa-4e74-a7b6-f90c95f59092-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.738600 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.738576 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed9d7305-2faa-4e74-a7b6-f90c95f59092-data-volume\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.738893 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.738871 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-config-volume\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.741700 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.741676 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-metrics-tls\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.741818 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.741791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed9d7305-2faa-4e74-a7b6-f90c95f59092-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.741952 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.741930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/546cdf59-c034-4ab6-a386-f8ddfe14aeb3-cert\") pod \"ingress-canary-ncsgj\" (UID: \"546cdf59-c034-4ab6-a386-f8ddfe14aeb3\") " pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.750189 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.750160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7vz\" (UniqueName: \"kubernetes.io/projected/546cdf59-c034-4ab6-a386-f8ddfe14aeb3-kube-api-access-mj7vz\") pod \"ingress-canary-ncsgj\" (UID: \"546cdf59-c034-4ab6-a386-f8ddfe14aeb3\") " pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.750878 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.750852 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-694hc\" (UniqueName: \"kubernetes.io/projected/ed9d7305-2faa-4e74-a7b6-f90c95f59092-kube-api-access-694hc\") pod \"insights-runtime-extractor-ndt2d\" (UID: \"ed9d7305-2faa-4e74-a7b6-f90c95f59092\") " pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:34.751527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.751509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8hv\" (UniqueName: \"kubernetes.io/projected/537a9c5d-b2f4-4410-8a28-ff7fb59f5b96-kube-api-access-6v8hv\") pod \"dns-default-gwf96\" (UID: \"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96\") " pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.783518 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.783485 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:34.806593 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.806551 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ncsgj" Aug 04 12:28:34.817499 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:34.817476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ndt2d" Aug 04 12:28:35.009067 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.009034 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gwf96"] Aug 04 12:28:35.012700 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:35.012676 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537a9c5d_b2f4_4410_8a28_ff7fb59f5b96.slice/crio-80698babf3a5cbea08bd8b5808cb819953e7777d1f6b116bb23ec8f358e1f112 WatchSource:0}: Error finding container 80698babf3a5cbea08bd8b5808cb819953e7777d1f6b116bb23ec8f358e1f112: Status 404 returned error can't find the container with id 80698babf3a5cbea08bd8b5808cb819953e7777d1f6b116bb23ec8f358e1f112 Aug 04 12:28:35.015616 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.015596 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ncsgj"] Aug 04 12:28:35.018795 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:35.018754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546cdf59_c034_4ab6_a386_f8ddfe14aeb3.slice/crio-f7b69a93b58a3bb5bd214cd49e36cad09dc1f3f47550ead2bbca7631561e6ddf WatchSource:0}: Error finding container f7b69a93b58a3bb5bd214cd49e36cad09dc1f3f47550ead2bbca7631561e6ddf: Status 404 returned error can't find the container with id f7b69a93b58a3bb5bd214cd49e36cad09dc1f3f47550ead2bbca7631561e6ddf Aug 04 12:28:35.027856 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.027829 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ndt2d"] Aug 04 12:28:35.030617 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:35.030589 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9d7305_2faa_4e74_a7b6_f90c95f59092.slice/crio-7313f4d4e970b2b8c7c474eae1363b0f57def23a43a23b74c0dbad981688f264 WatchSource:0}: Error finding container 7313f4d4e970b2b8c7c474eae1363b0f57def23a43a23b74c0dbad981688f264: Status 404 returned error can't find the container with id 7313f4d4e970b2b8c7c474eae1363b0f57def23a43a23b74c0dbad981688f264 Aug 04 12:28:35.767644 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.767606 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:35.768887 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.768845 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cff8cdbd6-xbfft"] Aug 04 12:28:35.778116 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.777488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.783586 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.783523 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 12:28:35.783701 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.783600 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 12:28:35.783979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.783832 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hhk5r\"" Aug 04 12:28:35.784070 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.784057 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 12:28:35.784264 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.784248 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 12:28:35.784500 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.784486 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 12:28:35.785209 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.785188 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 12:28:35.785422 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.785407 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 12:28:35.786319 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.785733 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cff8cdbd6-xbfft"] Aug 04 12:28:35.847712 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.847675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcmw\" (UniqueName: \"kubernetes.io/projected/d2133776-e891-410f-9988-a0e839e0cc21-kube-api-access-qwcmw\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.847904 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.847728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-console-config\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.847904 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.847797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-oauth-config\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.847904 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.847825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-serving-cert\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.847904 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.847887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-service-ca\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.848145 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.848002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-oauth-serving-cert\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.860278 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.860031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndt2d" event={"ID":"ed9d7305-2faa-4e74-a7b6-f90c95f59092","Type":"ContainerStarted","Data":"0c1de2fc63eebf14d9ca8a073938c66dc5e23d66c41138e244d876034d3f9c82"} Aug 04 12:28:35.860278 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.860298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndt2d" event={"ID":"ed9d7305-2faa-4e74-a7b6-f90c95f59092","Type":"ContainerStarted","Data":"7313f4d4e970b2b8c7c474eae1363b0f57def23a43a23b74c0dbad981688f264"} Aug 04 12:28:35.861423 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.861397 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ncsgj" event={"ID":"546cdf59-c034-4ab6-a386-f8ddfe14aeb3","Type":"ContainerStarted","Data":"f7b69a93b58a3bb5bd214cd49e36cad09dc1f3f47550ead2bbca7631561e6ddf"} Aug 04 12:28:35.862643 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.862616 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gwf96" event={"ID":"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96","Type":"ContainerStarted","Data":"80698babf3a5cbea08bd8b5808cb819953e7777d1f6b116bb23ec8f358e1f112"} Aug 04 12:28:35.866706 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.866683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:35.948463 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.948422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-service-ca\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.948664 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.948510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-oauth-serving-cert\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.948664 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.948547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcmw\" (UniqueName: \"kubernetes.io/projected/d2133776-e891-410f-9988-a0e839e0cc21-kube-api-access-qwcmw\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.948664 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.948579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-console-config\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.948664 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.948623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-oauth-config\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.948664 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.948652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-serving-cert\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.950044 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.949984 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-oauth-serving-cert\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.950044 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.949984 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-console-config\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.950237 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.950099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-service-ca\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.953208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.953182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-oauth-config\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.953339 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.953245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-serving-cert\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:35.959884 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:35.959833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcmw\" (UniqueName: \"kubernetes.io/projected/d2133776-e891-410f-9988-a0e839e0cc21-kube-api-access-qwcmw\") pod \"console-cff8cdbd6-xbfft\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:36.091058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:36.091023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:37.178111 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.178015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:37.282420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.282382 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:37.402216 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.402179 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:37.448677 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.448598 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-j4x4b"] Aug 04 12:28:37.471733 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.471701 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-j4x4b"] Aug 04 12:28:37.471894 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.471829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.475670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.475629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:37.476514 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.476493 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 12:28:37.476514 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.476519 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7vptk\"" Aug 04 12:28:37.476679 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.476532 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 12:28:37.476679 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.476566 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 12:28:37.476679 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.476627 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 12:28:37.476860 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.476841 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 12:28:37.562755 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.562728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0be2614a-2d46-476a-9a62-e2ee371bcf99-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.562915 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.562783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.562915 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.562823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9srw\" (UniqueName: \"kubernetes.io/projected/0be2614a-2d46-476a-9a62-e2ee371bcf99-kube-api-access-x9srw\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.562915 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.562846 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.663227 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.663189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.663405 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.663253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0be2614a-2d46-476a-9a62-e2ee371bcf99-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.663405 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.663298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.663405 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.663327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9srw\" (UniqueName: \"kubernetes.io/projected/0be2614a-2d46-476a-9a62-e2ee371bcf99-kube-api-access-x9srw\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.663405 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:37.663342 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 04 12:28:37.663617 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:37.663413 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-tls podName:0be2614a-2d46-476a-9a62-e2ee371bcf99 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:38.16339605 +0000 UTC m=+49.118991822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-j4x4b" (UID: "0be2614a-2d46-476a-9a62-e2ee371bcf99") : secret "prometheus-operator-tls" not found Aug 04 12:28:37.663938 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.663915 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0be2614a-2d46-476a-9a62-e2ee371bcf99-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.665907 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.665886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.673857 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.673838 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9srw\" (UniqueName: \"kubernetes.io/projected/0be2614a-2d46-476a-9a62-e2ee371bcf99-kube-api-access-x9srw\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:37.759173 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.759069 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:37.961397 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:37.961211 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:38.057796 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.057766 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:38.070054 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.069460 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cff8cdbd6-xbfft"] Aug 04 12:28:38.072734 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:38.072710 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2133776_e891_410f_9988_a0e839e0cc21.slice/crio-f6de99e2fe2c0d17f787c748f21511fc93241a396cae2c13be554691362e2503 WatchSource:0}: Error finding container f6de99e2fe2c0d17f787c748f21511fc93241a396cae2c13be554691362e2503: Status 404 returned error can't find the container with id f6de99e2fe2c0d17f787c748f21511fc93241a396cae2c13be554691362e2503 Aug 04 12:28:38.169737 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.169705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:38.173950 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.173922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be2614a-2d46-476a-9a62-e2ee371bcf99-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-j4x4b\" (UID: \"0be2614a-2d46-476a-9a62-e2ee371bcf99\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:38.383226 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.383184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" Aug 04 12:28:38.527527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.527497 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-j4x4b"] Aug 04 12:28:38.530924 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:38.530901 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be2614a_2d46_476a_9a62_e2ee371bcf99.slice/crio-921608f5a7689334c660713352323528273bdf068123cb2dc59ada81f90644d6 WatchSource:0}: Error finding container 921608f5a7689334c660713352323528273bdf068123cb2dc59ada81f90644d6: Status 404 returned error can't find the container with id 921608f5a7689334c660713352323528273bdf068123cb2dc59ada81f90644d6 Aug 04 12:28:38.876681 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.876644 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndt2d" event={"ID":"ed9d7305-2faa-4e74-a7b6-f90c95f59092","Type":"ContainerStarted","Data":"3d0d9c6ef7b716c8def68bec88194b1fae730c5000459b7a9b0ba626afef55bb"} Aug 04 12:28:38.878876 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.878837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gwf96" event={"ID":"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96","Type":"ContainerStarted","Data":"3f185f2ab6e0e514b0c23824603b5ae597a00a27d9612724f9015b4d6c69f443"} Aug 04 12:28:38.879026 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.878898 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gwf96" event={"ID":"537a9c5d-b2f4-4410-8a28-ff7fb59f5b96","Type":"ContainerStarted","Data":"b4dbd21beec8033fbf511860eb9abec8854e9e25617e3b16ca32500935d1bfde"} Aug 04 12:28:38.879026 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.878970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:38.880343 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.880315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cff8cdbd6-xbfft" event={"ID":"d2133776-e891-410f-9988-a0e839e0cc21","Type":"ContainerStarted","Data":"f6de99e2fe2c0d17f787c748f21511fc93241a396cae2c13be554691362e2503"} Aug 04 12:28:38.881589 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.881556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" event={"ID":"0be2614a-2d46-476a-9a62-e2ee371bcf99","Type":"ContainerStarted","Data":"921608f5a7689334c660713352323528273bdf068123cb2dc59ada81f90644d6"} Aug 04 12:28:38.883159 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.883137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ncsgj" event={"ID":"546cdf59-c034-4ab6-a386-f8ddfe14aeb3","Type":"ContainerStarted","Data":"cd4f7acdb6856e4f65cbba845f6b93f94e65193be0a76941a4a7a2a43d75d040"} Aug 04 12:28:38.902295 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.902232 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gwf96" podStartSLOduration=2.017972688 podStartE2EDuration="4.902216203s" totalImagesPullingTime="2.884243515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:38.90103142 +0000 UTC m=+49.856627214" watchObservedRunningTime="2026-08-04 12:28:38.902216203 +0000 UTC m=+49.857811997" Aug 04 12:28:38.920993 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:38.920935 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ncsgj" podStartSLOduration=2.035431045 podStartE2EDuration="4.920918965s" totalImagesPullingTime="2.88548792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:38.920283542 +0000 UTC m=+49.875879337" watchObservedRunningTime="2026-08-04 12:28:38.920918965 +0000 UTC m=+49.876514758" Aug 04 12:28:39.686601 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:39.686561 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:39.774666 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:39.774637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:39.882998 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:39.882936 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:39.980264 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:39.980185 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:40.232356 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:40.232269 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:42.414565 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.414403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:42.544637 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.544596 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:42.645934 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.645904 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:42.898516 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.898487 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cff8cdbd6-xbfft" event={"ID":"d2133776-e891-410f-9988-a0e839e0cc21","Type":"ContainerStarted","Data":"f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15"} Aug 04 12:28:42.900309 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.900281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" event={"ID":"0be2614a-2d46-476a-9a62-e2ee371bcf99","Type":"ContainerStarted","Data":"e05b593702f73568861e63ab74c7cd721f4b5bfb671d869ffe34651b0830953f"} Aug 04 12:28:42.900440 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.900323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" event={"ID":"0be2614a-2d46-476a-9a62-e2ee371bcf99","Type":"ContainerStarted","Data":"f9515e774642ccaa184e1e773a2dec64e8378f6513fac5b5487de6e37604fd96"} Aug 04 12:28:42.902407 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.902386 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndt2d" event={"ID":"ed9d7305-2faa-4e74-a7b6-f90c95f59092","Type":"ContainerStarted","Data":"19f03881a4f371795bedc85d5d92fa5f42a81d9d31648cde88f0e7fe3bf71a7e"} Aug 04 12:28:42.920849 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.920808 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cff8cdbd6-xbfft" podStartSLOduration=3.95177172 podStartE2EDuration="7.920794734s" totalImagesPullingTime="3.969023014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:42.920469182 +0000 UTC m=+53.876064979" watchObservedRunningTime="2026-08-04 12:28:42.920794734 +0000 UTC m=+53.876390527" Aug 04 12:28:42.941725 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:42.941676 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ndt2d" podStartSLOduration=2.169254387 podStartE2EDuration="8.941660589s" totalImagesPullingTime="6.772406202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:28:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:42.941200834 +0000 UTC m=+53.896796628" watchObservedRunningTime="2026-08-04 12:28:42.941660589 +0000 UTC m=+53.897256423" Aug 04 12:28:43.935584 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:43.935557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:43.994136 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:43.994079 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:44.598501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.598458 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:44.852352 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.852239 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-j4x4b" podStartSLOduration=4.347383711 podStartE2EDuration="7.852222637s" totalImagesPullingTime="3.504838926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:42.958480694 +0000 UTC m=+53.914076556" watchObservedRunningTime="2026-08-04 12:28:44.852222637 +0000 UTC m=+55.807818431" Aug 04 12:28:44.853068 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.853046 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8"] Aug 04 12:28:44.855160 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.855145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:44.860182 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.860163 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 12:28:44.860259 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.860182 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 12:28:44.860615 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.860599 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mmkzs\"" Aug 04 12:28:44.865319 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.865301 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f"] Aug 04 12:28:44.867233 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.867216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.872086 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.872064 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 04 12:28:44.872384 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.872367 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 04 12:28:44.872384 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.872379 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ztm6n\"" Aug 04 12:28:44.872508 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.872418 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 04 12:28:44.880175 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.880155 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8"] Aug 04 12:28:44.884230 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.884198 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f"] Aug 04 12:28:44.886258 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.886230 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:44.902136 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.902096 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5j7h7"] Aug 04 12:28:44.904295 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.904281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:44.907760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.907742 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 12:28:44.908430 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.908413 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gjl6d\"" Aug 04 12:28:44.908547 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.908533 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 12:28:44.908640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.908624 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 12:28:44.923960 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.923938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150af977-817a-4471-98fe-37818dce152a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:44.924069 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.923969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c66317-9eda-412f-85e2-9415e04e829c-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.924069 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.923997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/15c66317-9eda-412f-85e2-9415e04e829c-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.924069 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.924197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150af977-817a-4471-98fe-37818dce152a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:44.924197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gsp\" (UniqueName: \"kubernetes.io/projected/15c66317-9eda-412f-85e2-9415e04e829c-kube-api-access-62gsp\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.924197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748mx\" (UniqueName: \"kubernetes.io/projected/150af977-817a-4471-98fe-37818dce152a-kube-api-access-748mx\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:44.924197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.924313 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:44.924313 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.924277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150af977-817a-4471-98fe-37818dce152a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:44.973107 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.973073 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:44.973791 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.973771 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bbbb79648-kpdx7"] Aug 04 12:28:44.975609 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.975591 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:44.988164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.988141 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 12:28:44.995396 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:44.995371 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbbb79648-kpdx7"] Aug 04 12:28:45.025432 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.025432 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-trusted-ca-bundle\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-metrics-client-ca\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5vh\" (UniqueName: \"kubernetes.io/projected/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-kube-api-access-6d5vh\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150af977-817a-4471-98fe-37818dce152a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-service-ca\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.025670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150af977-817a-4471-98fe-37818dce152a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c66317-9eda-412f-85e2-9415e04e829c-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-tls\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/15c66317-9eda-412f-85e2-9415e04e829c-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-config\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.025991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.025893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-root\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-sys\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-wtmp\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/15c66317-9eda-412f-85e2-9415e04e829c-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-oauth-serving-cert\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150af977-817a-4471-98fe-37818dce152a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-textfile\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.026361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-oauth-config\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.026678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-serving-cert\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.026678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/150af977-817a-4471-98fe-37818dce152a-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.026678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.026678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gsp\" (UniqueName: \"kubernetes.io/projected/15c66317-9eda-412f-85e2-9415e04e829c-kube-api-access-62gsp\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.026678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zx2d\" (UniqueName: \"kubernetes.io/projected/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-kube-api-access-2zx2d\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.026678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.026511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748mx\" (UniqueName: \"kubernetes.io/projected/150af977-817a-4471-98fe-37818dce152a-kube-api-access-748mx\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.027191 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.027171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c66317-9eda-412f-85e2-9415e04e829c-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.029709 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.029686 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/150af977-817a-4471-98fe-37818dce152a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.029709 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.029706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/150af977-817a-4471-98fe-37818dce152a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.029914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.029893 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.029951 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.029939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15c66317-9eda-412f-85e2-9415e04e829c-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.040579 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.040554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-748mx\" (UniqueName: \"kubernetes.io/projected/150af977-817a-4471-98fe-37818dce152a-kube-api-access-748mx\") pod \"openshift-state-metrics-6bb4d7b4dd-wj8b8\" (UID: \"150af977-817a-4471-98fe-37818dce152a\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.041272 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.041249 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gsp\" (UniqueName: \"kubernetes.io/projected/15c66317-9eda-412f-85e2-9415e04e829c-kube-api-access-62gsp\") pod \"kube-state-metrics-6bd9664c4-gfz2f\" (UID: \"15c66317-9eda-412f-85e2-9415e04e829c\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.071320 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.071284 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.127039 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.126967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-wtmp\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127039 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-oauth-serving-cert\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127039 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-textfile\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-oauth-config\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-serving-cert\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zx2d\" (UniqueName: \"kubernetes.io/projected/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-kube-api-access-2zx2d\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-trusted-ca-bundle\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-metrics-client-ca\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5vh\" (UniqueName: \"kubernetes.io/projected/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-kube-api-access-6d5vh\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-wtmp\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-service-ca\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-tls\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-config\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-root\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-sys\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-textfile\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127530 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-sys\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:45.127606 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 12:28:45.127787 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:28:45.127672 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-tls podName:462251bf-8039-4bfe-b33d-4fcf0a94e3f0 nodeName:}" failed. No retries permitted until 2026-08-04 12:28:45.627654415 +0000 UTC m=+56.583250189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-tls") pod "node-exporter-5j7h7" (UID: "462251bf-8039-4bfe-b33d-4fcf0a94e3f0") : secret "node-exporter-tls" not found Aug 04 12:28:45.128342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-oauth-serving-cert\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.128342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-metrics-client-ca\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.128342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.127997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.128342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.128017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-root\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.128342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.128288 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-trusted-ca-bundle\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.128677 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.128659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-service-ca\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.128751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.128720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-config\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.129951 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.129920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-serving-cert\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.130111 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.130089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-oauth-config\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.130184 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.130146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.146568 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.146540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5vh\" (UniqueName: \"kubernetes.io/projected/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-kube-api-access-6d5vh\") pod \"console-7bbbb79648-kpdx7\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.150294 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.150276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zx2d\" (UniqueName: \"kubernetes.io/projected/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-kube-api-access-2zx2d\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.164168 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.164139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" Aug 04 12:28:45.175833 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.175809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" Aug 04 12:28:45.235093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.234790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.284577 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.284529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:45.314747 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.314723 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8"] Aug 04 12:28:45.317134 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:45.317091 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150af977_817a_4471_98fe_37818dce152a.slice/crio-b34565123fbb75ee9d9dec1cd714a2a217d8dfd7fc4127e2571caef6c7624c71 WatchSource:0}: Error finding container b34565123fbb75ee9d9dec1cd714a2a217d8dfd7fc4127e2571caef6c7624c71: Status 404 returned error can't find the container with id b34565123fbb75ee9d9dec1cd714a2a217d8dfd7fc4127e2571caef6c7624c71 Aug 04 12:28:45.324816 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.324782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f"] Aug 04 12:28:45.328491 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.328468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.415055 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.415028 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.434210 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.434185 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bbbb79648-kpdx7"] Aug 04 12:28:45.437505 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:45.437481 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60e4ff1_c0e6_4981_aa47_c65141dd96a6.slice/crio-441dd197431406d8ff9ea40bcb1af7a1c8dfa92588aadf39fc98124513d67a6e WatchSource:0}: Error finding container 441dd197431406d8ff9ea40bcb1af7a1c8dfa92588aadf39fc98124513d67a6e: Status 404 returned error can't find the container with id 441dd197431406d8ff9ea40bcb1af7a1c8dfa92588aadf39fc98124513d67a6e Aug 04 12:28:45.554817 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.554765 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.631185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.631093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-tls\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.634698 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.634679 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/462251bf-8039-4bfe-b33d-4fcf0a94e3f0-node-exporter-tls\") pod \"node-exporter-5j7h7\" (UID: \"462251bf-8039-4bfe-b33d-4fcf0a94e3f0\") " pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.812870 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.812842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5j7h7" Aug 04 12:28:45.823424 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:45.823398 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462251bf_8039_4bfe_b33d_4fcf0a94e3f0.slice/crio-3d1b29cfbec37e6ab2ac0e1c769a288a7b3ffcbd508ad31d3cd5f2f9449edd4c WatchSource:0}: Error finding container 3d1b29cfbec37e6ab2ac0e1c769a288a7b3ffcbd508ad31d3cd5f2f9449edd4c: Status 404 returned error can't find the container with id 3d1b29cfbec37e6ab2ac0e1c769a288a7b3ffcbd508ad31d3cd5f2f9449edd4c Aug 04 12:28:45.833541 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.833512 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.916874 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.916787 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" event={"ID":"150af977-817a-4471-98fe-37818dce152a","Type":"ContainerStarted","Data":"8827cf63bc4d33246e70d3e934a12cd2d5982122ad55ddeba7d64098db961b57"} Aug 04 12:28:45.916874 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.916858 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" event={"ID":"150af977-817a-4471-98fe-37818dce152a","Type":"ContainerStarted","Data":"927fe988a62ecf82f7aeb3f62973a66c833c629d57a3557552b0fd567197799d"} Aug 04 12:28:45.917058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.916887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" event={"ID":"150af977-817a-4471-98fe-37818dce152a","Type":"ContainerStarted","Data":"b34565123fbb75ee9d9dec1cd714a2a217d8dfd7fc4127e2571caef6c7624c71"} Aug 04 12:28:45.918072 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.918042 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" event={"ID":"15c66317-9eda-412f-85e2-9415e04e829c","Type":"ContainerStarted","Data":"872d27bfe9ea9b15defc4ffcee81e2993bf38822527ea8ca5bd918c68ae29241"} Aug 04 12:28:45.919220 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.919194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j7h7" event={"ID":"462251bf-8039-4bfe-b33d-4fcf0a94e3f0","Type":"ContainerStarted","Data":"3d1b29cfbec37e6ab2ac0e1c769a288a7b3ffcbd508ad31d3cd5f2f9449edd4c"} Aug 04 12:28:45.920519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.920498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbbb79648-kpdx7" event={"ID":"c60e4ff1-c0e6-4981-aa47-c65141dd96a6","Type":"ContainerStarted","Data":"bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3"} Aug 04 12:28:45.920587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.920542 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbbb79648-kpdx7" event={"ID":"c60e4ff1-c0e6-4981-aa47-c65141dd96a6","Type":"ContainerStarted","Data":"441dd197431406d8ff9ea40bcb1af7a1c8dfa92588aadf39fc98124513d67a6e"} Aug 04 12:28:45.932577 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.932557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:45.944214 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.944197 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:28:45.947096 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.947083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:45.950373 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950350 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 12:28:45.950373 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950351 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 12:28:45.950634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950620 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 12:28:45.950709 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 12:28:45.950709 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950666 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 12:28:45.950928 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950913 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 12:28:45.951005 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950947 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 12:28:45.951058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.950947 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z7xzm\"" Aug 04 12:28:45.952958 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.952891 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bbbb79648-kpdx7" podStartSLOduration=1.952878098 podStartE2EDuration="1.952878098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:45.951307369 +0000 UTC m=+56.906903164" watchObservedRunningTime="2026-08-04 12:28:45.952878098 +0000 UTC m=+56.908473883" Aug 04 12:28:45.953539 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.953521 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 12:28:45.953626 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.953593 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 12:28:45.967954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:45.967911 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:28:46.032907 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.032872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:46.034267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-out\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034626 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034902 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bgf\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-kube-api-access-m9bgf\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034948 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.034948 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.035045 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.034995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-web-config\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.035244 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.035215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.091656 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.091617 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:46.091816 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.091720 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:28:46.093352 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.093325 2578 patch_prober.go:28] interesting pod/console-cff8cdbd6-xbfft container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Aug 04 12:28:46.093487 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.093375 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cff8cdbd6-xbfft" podUID="d2133776-e891-410f-9988-a0e839e0cc21" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Aug 04 12:28:46.136113 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bgf\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-kube-api-access-m9bgf\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.136320 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.136391 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.136450 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-web-config\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.136450 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.136590 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-out\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.136645 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.137455 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.137455 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.136986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.137455 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.137075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.137455 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.137115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.137455 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.137189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.137455 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.137260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.138043 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.137641 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.138113 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.137969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.138993 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.138926 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.140811 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.140783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.140883 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.140789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-web-config\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.142112 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.142058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-out\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.142858 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.142834 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.143332 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.143311 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.144061 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.143714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.144061 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.143944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.144061 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.144021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.147414 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.147386 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-volume\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.149748 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.149724 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bgf\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-kube-api-access-m9bgf\") pod \"alertmanager-main-0\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.257738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.257649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:28:46.429311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.429070 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:28:46.717971 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:46.717936 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aae3095_5bb2_4bd4_8e8f_d109436b765f.slice/crio-2d50f7903da26a0cc7e2b71dc5527556a240a73ef70f2b5d850accd4cf2bf5ce WatchSource:0}: Error finding container 2d50f7903da26a0cc7e2b71dc5527556a240a73ef70f2b5d850accd4cf2bf5ce: Status 404 returned error can't find the container with id 2d50f7903da26a0cc7e2b71dc5527556a240a73ef70f2b5d850accd4cf2bf5ce Aug 04 12:28:46.900090 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.900050 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:46.930564 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.930524 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"2d50f7903da26a0cc7e2b71dc5527556a240a73ef70f2b5d850accd4cf2bf5ce"} Aug 04 12:28:46.947622 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.947589 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-584c6647fc-tdbpc"] Aug 04 12:28:46.950419 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.950395 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:46.953474 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.953450 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 12:28:46.953720 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.953703 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 12:28:46.954019 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.953990 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2c5fq\"" Aug 04 12:28:46.954182 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.954032 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 12:28:46.954272 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.954251 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cpccfv5ic8a1a\"" Aug 04 12:28:46.955065 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.955048 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 12:28:46.955238 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.955221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 12:28:46.969532 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.969475 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-584c6647fc-tdbpc"] Aug 04 12:28:46.976603 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:46.976577 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:47.046952 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.046848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgr5\" (UniqueName: \"kubernetes.io/projected/f35733d2-ce07-4b1d-ab43-bd6e9b437271-kube-api-access-mpgr5\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35733d2-ce07-4b1d-ab43-bd6e9b437271-metrics-client-ca\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-tls\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.047995 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.047882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-grpc-tls\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.064152 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.064107 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:47.148902 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.148842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.148918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-tls\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.148957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-grpc-tls\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.149020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.149047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149292 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.149110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgr5\" (UniqueName: \"kubernetes.io/projected/f35733d2-ce07-4b1d-ab43-bd6e9b437271-kube-api-access-mpgr5\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149292 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.149167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35733d2-ce07-4b1d-ab43-bd6e9b437271-metrics-client-ca\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.149292 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.149230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.151681 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.151622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35733d2-ce07-4b1d-ab43-bd6e9b437271-metrics-client-ca\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.153269 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.153245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.153843 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.153824 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-grpc-tls\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.154919 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.154868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.155471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.155420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-tls\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.155471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.155453 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.155921 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.155875 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f35733d2-ce07-4b1d-ab43-bd6e9b437271-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.171015 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.170984 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgr5\" (UniqueName: \"kubernetes.io/projected/f35733d2-ce07-4b1d-ab43-bd6e9b437271-kube-api-access-mpgr5\") pod \"thanos-querier-584c6647fc-tdbpc\" (UID: \"f35733d2-ce07-4b1d-ab43-bd6e9b437271\") " pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.262145 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.262021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:47.484342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.483213 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-584c6647fc-tdbpc"] Aug 04 12:28:47.538398 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:47.538289 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35733d2_ce07_4b1d_ab43_bd6e9b437271.slice/crio-c80e57fbe7361227e27d83086f534146d47e6c8d56d8455dd2dd03005479e7c7 WatchSource:0}: Error finding container c80e57fbe7361227e27d83086f534146d47e6c8d56d8455dd2dd03005479e7c7: Status 404 returned error can't find the container with id c80e57fbe7361227e27d83086f534146d47e6c8d56d8455dd2dd03005479e7c7 Aug 04 12:28:47.752475 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.752437 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:47.853156 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.853109 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:47.939256 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.939211 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" event={"ID":"15c66317-9eda-412f-85e2-9415e04e829c","Type":"ContainerStarted","Data":"64909698f9171f315e4381424d1fecf723cc0de2a6acef94371a8243a3d42c8a"} Aug 04 12:28:47.939437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.939279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" event={"ID":"15c66317-9eda-412f-85e2-9415e04e829c","Type":"ContainerStarted","Data":"77a9e0657fc34067a9856ae35442152127b53e8b76906c7587ce3c3fad8caf44"} Aug 04 12:28:47.939437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.939312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" event={"ID":"15c66317-9eda-412f-85e2-9415e04e829c","Type":"ContainerStarted","Data":"d9222e9828ea6becc50c686bbe850f84dc40ac1605bd79cec5bf78e7b5e483fd"} Aug 04 12:28:47.940965 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.940861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"c80e57fbe7361227e27d83086f534146d47e6c8d56d8455dd2dd03005479e7c7"} Aug 04 12:28:47.942673 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.942646 2578 generic.go:356] "Generic (PLEG): container finished" podID="462251bf-8039-4bfe-b33d-4fcf0a94e3f0" containerID="35d2757fda8cbbc3c1a6f3ccca9ee470ae7f92460e5082496d10ba1bb5251fbe" exitCode=0 Aug 04 12:28:47.942793 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.942744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j7h7" event={"ID":"462251bf-8039-4bfe-b33d-4fcf0a94e3f0","Type":"ContainerDied","Data":"35d2757fda8cbbc3c1a6f3ccca9ee470ae7f92460e5082496d10ba1bb5251fbe"} Aug 04 12:28:47.945436 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.945413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" event={"ID":"150af977-817a-4471-98fe-37818dce152a","Type":"ContainerStarted","Data":"7a66f1157eab4cc1f65785e12d0cfd9c3a5c1fe62798db7373e33dd48e893318"} Aug 04 12:28:47.962371 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.962328 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-gfz2f" podStartSLOduration=2.314252944 podStartE2EDuration="3.962312468s" totalImagesPullingTime="1.648059524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:47.961530354 +0000 UTC m=+58.917126164" watchObservedRunningTime="2026-08-04 12:28:47.962312468 +0000 UTC m=+58.917908261" Aug 04 12:28:47.993401 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:47.993355 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-wj8b8" podStartSLOduration=2.458712396 podStartE2EDuration="3.993338516s" totalImagesPullingTime="1.53462612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:47.986758424 +0000 UTC m=+58.942354218" watchObservedRunningTime="2026-08-04 12:28:47.993338516 +0000 UTC m=+58.948934310" Aug 04 12:28:48.026621 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.026597 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:48.114933 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.114803 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:48.217391 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.217358 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:48.316408 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.316377 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:48.558812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.558557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:48.659708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.659677 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:48.890173 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.890141 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gwf96" Aug 04 12:28:48.952442 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.952290 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7" exitCode=0 Aug 04 12:28:48.952442 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.952345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7"} Aug 04 12:28:48.956702 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.956653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j7h7" event={"ID":"462251bf-8039-4bfe-b33d-4fcf0a94e3f0","Type":"ContainerStarted","Data":"40b478f58691358103a6767d6ecc838c40ef7dcc434e02dab5243c9f698f69e8"} Aug 04 12:28:48.956812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:48.956711 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5j7h7" event={"ID":"462251bf-8039-4bfe-b33d-4fcf0a94e3f0","Type":"ContainerStarted","Data":"d97c6104366eb8987b64f06d42ef35fd76290ecc3b936df1280d473a0a008a77"} Aug 04 12:28:49.010552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.010488 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5j7h7" podStartSLOduration=3.449179092 podStartE2EDuration="5.010462808s" totalImagesPullingTime="1.477807979s" totalInitContainerRuntime="83.475737ms" isStatefulPod=true podCreationTimestamp="2026-08-04 12:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:49.008464267 +0000 UTC m=+59.964060062" watchObservedRunningTime="2026-08-04 12:28:49.010462808 +0000 UTC m=+59.966058603" Aug 04 12:28:49.313317 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.313226 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d76c6449f-jp7q7"] Aug 04 12:28:49.316195 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.316163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.319992 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.319750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 12:28:49.319992 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.319906 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 12:28:49.320197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.320056 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7sqeqq0dipss6\"" Aug 04 12:28:49.320197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.320139 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 12:28:49.321209 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.321193 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 12:28:49.321294 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.321240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bn672\"" Aug 04 12:28:49.333022 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.332995 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d76c6449f-jp7q7"] Aug 04 12:28:49.371246 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.371215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-secret-metrics-server-client-certs\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.371246 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.371247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6191714-9ae7-42f7-acdf-529da820a389-metrics-server-audit-profiles\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.371504 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.371305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-client-ca-bundle\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.371504 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.371326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-secret-metrics-server-tls\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.371504 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.371365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6191714-9ae7-42f7-acdf-529da820a389-audit-log\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.372219 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.372172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6191714-9ae7-42f7-acdf-529da820a389-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.372560 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.372535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ncm\" (UniqueName: \"kubernetes.io/projected/f6191714-9ae7-42f7-acdf-529da820a389-kube-api-access-59ncm\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473566 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ncm\" (UniqueName: \"kubernetes.io/projected/f6191714-9ae7-42f7-acdf-529da820a389-kube-api-access-59ncm\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473688 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-secret-metrics-server-client-certs\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6191714-9ae7-42f7-acdf-529da820a389-metrics-server-audit-profiles\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473751 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-client-ca-bundle\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473836 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-secret-metrics-server-tls\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473836 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6191714-9ae7-42f7-acdf-529da820a389-audit-log\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.473943 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.473920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6191714-9ae7-42f7-acdf-529da820a389-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.474226 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.474204 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f6191714-9ae7-42f7-acdf-529da820a389-audit-log\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.477825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.477784 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 12:28:49.477939 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.477801 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7sqeqq0dipss6\"" Aug 04 12:28:49.477939 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.477809 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 12:28:49.478068 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.477844 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 12:28:49.478068 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.477865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 12:28:49.485277 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.484962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ncm\" (UniqueName: \"kubernetes.io/projected/f6191714-9ae7-42f7-acdf-529da820a389-kube-api-access-59ncm\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.485421 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.485398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f6191714-9ae7-42f7-acdf-529da820a389-metrics-server-audit-profiles\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.485805 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.485783 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6191714-9ae7-42f7-acdf-529da820a389-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.486976 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.486948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-secret-metrics-server-tls\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.487541 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.487520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-client-ca-bundle\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.487778 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.487762 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f6191714-9ae7-42f7-acdf-529da820a389-secret-metrics-server-client-certs\") pod \"metrics-server-d76c6449f-jp7q7\" (UID: \"f6191714-9ae7-42f7-acdf-529da820a389\") " pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.626416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.626381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:49.631338 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.631308 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bn672\"" Aug 04 12:28:49.639236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.639212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:28:49.652864 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.652836 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz"] Aug 04 12:28:49.655569 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.655545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.659494 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.659165 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 12:28:49.659494 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.659171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t2k28\"" Aug 04 12:28:49.670670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.670598 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz"] Aug 04 12:28:49.718184 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.718153 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:49.776732 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.776699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqghn\" (UniqueName: \"kubernetes.io/projected/fb3b212a-694c-4fd0-959c-ae6b3d3797f4-kube-api-access-vqghn\") pod \"monitoring-plugin-6dffbbd67-2l5jz\" (UID: \"fb3b212a-694c-4fd0-959c-ae6b3d3797f4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.776869 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.776804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb3b212a-694c-4fd0-959c-ae6b3d3797f4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2l5jz\" (UID: \"fb3b212a-694c-4fd0-959c-ae6b3d3797f4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.801150 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.801100 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d76c6449f-jp7q7"] Aug 04 12:28:49.806288 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:49.806257 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6191714_9ae7_42f7_acdf_529da820a389.slice/crio-30889650fdcd4d42c86a02458a899339b2083e31adbbcc54359330229e9a92d5 WatchSource:0}: Error finding container 30889650fdcd4d42c86a02458a899339b2083e31adbbcc54359330229e9a92d5: Status 404 returned error can't find the container with id 30889650fdcd4d42c86a02458a899339b2083e31adbbcc54359330229e9a92d5 Aug 04 12:28:49.877956 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.877919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb3b212a-694c-4fd0-959c-ae6b3d3797f4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2l5jz\" (UID: \"fb3b212a-694c-4fd0-959c-ae6b3d3797f4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.878181 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.877999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqghn\" (UniqueName: \"kubernetes.io/projected/fb3b212a-694c-4fd0-959c-ae6b3d3797f4-kube-api-access-vqghn\") pod \"monitoring-plugin-6dffbbd67-2l5jz\" (UID: \"fb3b212a-694c-4fd0-959c-ae6b3d3797f4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.881203 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.881176 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fb3b212a-694c-4fd0-959c-ae6b3d3797f4-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-2l5jz\" (UID: \"fb3b212a-694c-4fd0-959c-ae6b3d3797f4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.894742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.894717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqghn\" (UniqueName: \"kubernetes.io/projected/fb3b212a-694c-4fd0-959c-ae6b3d3797f4-kube-api-access-vqghn\") pod \"monitoring-plugin-6dffbbd67-2l5jz\" (UID: \"fb3b212a-694c-4fd0-959c-ae6b3d3797f4\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:49.936542 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.936502 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:49.962264 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.962226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" event={"ID":"f6191714-9ae7-42f7-acdf-529da820a389","Type":"ContainerStarted","Data":"30889650fdcd4d42c86a02458a899339b2083e31adbbcc54359330229e9a92d5"} Aug 04 12:28:49.965074 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.965044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"816d1104851c376186fa70597b93945938abc7b0f920591d78a3001d1a17e96e"} Aug 04 12:28:49.965238 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.965137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"8eef5f15e0bf35d50eeb9ab4ab309a5cc22ac31775fd95a9729c3d1a18dba306"} Aug 04 12:28:49.965238 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.965179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"8bace4cafbf8aab05654b7f7219f1e99297237ffc60a58ecc7105e781cd34a87"} Aug 04 12:28:49.966457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:49.966438 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:50.037014 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:50.036859 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:50.126262 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:50.126232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz"] Aug 04 12:28:50.128182 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:50.128032 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3b212a_694c_4fd0_959c_ae6b3d3797f4.slice/crio-20a994d57376b9d5e7c879b14dd1b75b0db06980f5a87d3b1e71e2af257f50d0 WatchSource:0}: Error finding container 20a994d57376b9d5e7c879b14dd1b75b0db06980f5a87d3b1e71e2af257f50d0: Status 404 returned error can't find the container with id 20a994d57376b9d5e7c879b14dd1b75b0db06980f5a87d3b1e71e2af257f50d0 Aug 04 12:28:50.144649 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:50.144624 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:50.815853 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:50.815823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qch7k" Aug 04 12:28:50.957603 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:50.957568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:50.970397 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:50.970361 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" event={"ID":"fb3b212a-694c-4fd0-959c-ae6b3d3797f4","Type":"ContainerStarted","Data":"20a994d57376b9d5e7c879b14dd1b75b0db06980f5a87d3b1e71e2af257f50d0"} Aug 04 12:28:51.057340 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.057308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:51.158600 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.158563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:51.399857 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.399576 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:28:51.404236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.404212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.409584 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.409562 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 04 12:28:51.410042 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.410014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2bai1au17209d\"" Aug 04 12:28:51.410285 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.410269 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 04 12:28:51.410500 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.410484 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 04 12:28:51.410948 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.410930 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 04 12:28:51.411042 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.410975 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r4fwd\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411160 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411298 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411357 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411493 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411690 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411897 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.411995 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 04 12:28:51.413221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.412095 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 04 12:28:51.415162 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.415107 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 04 12:28:51.425687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.425658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:28:51.426680 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.425981 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 04 12:28:51.466339 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.466244 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.496989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.496964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdrx\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-kube-api-access-pwdrx\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.497718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.497264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598750 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdrx\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-kube-api-access-pwdrx\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598750 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.598988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.598981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599416 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.599892 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.599500 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.601587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.600032 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.601587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.600341 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.601587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.601269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.601835 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.601773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.602529 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.602499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.602955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.602931 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.602955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.602946 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.603279 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.603253 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.603543 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.603499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.607938 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.607914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.608877 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.608837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.608980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.608888 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.609480 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.609395 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.609480 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.609435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.609480 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.609452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.610048 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.610029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.610276 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.610260 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.610616 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.610584 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.611170 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.611151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdrx\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-kube-api-access-pwdrx\") pod \"prometheus-k8s-0\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.733928 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.733833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:28:51.998551 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:51.998522 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447"} Aug 04 12:28:52.003367 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.003326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987"} Aug 04 12:28:52.008653 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.008525 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"e0f5e1e5c54765116d8f9241af5e76397f373636ee3d8f8915041ef762338fb2"} Aug 04 12:28:52.008653 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.008601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"91da123d212676daa71a822e4d860deec75e3204030acd5caa6d2fdd1e714abb"} Aug 04 12:28:52.022202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.020959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" event={"ID":"f6191714-9ae7-42f7-acdf-529da820a389","Type":"ContainerStarted","Data":"a7e731936b7251358a511bf31f3a4e3fca0fe21ece9c8bf785e88f2d5ea9e18b"} Aug 04 12:28:52.049091 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.046798 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" podStartSLOduration=1.529569027 podStartE2EDuration="3.046777484s" totalImagesPullingTime="1.517208457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:52.045816914 +0000 UTC m=+63.001412709" watchObservedRunningTime="2026-08-04 12:28:52.046777484 +0000 UTC m=+63.002373280" Aug 04 12:28:52.106663 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.106585 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:28:52.122353 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:52.122322 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e4fe5e_5491_4c35_b219_82cc3a217ca8.slice/crio-c8ccb0402ff1a2478db658241944d4de401b1251283a755253c78898c1e0e30d WatchSource:0}: Error finding container c8ccb0402ff1a2478db658241944d4de401b1251283a755253c78898c1e0e30d: Status 404 returned error can't find the container with id c8ccb0402ff1a2478db658241944d4de401b1251283a755253c78898c1e0e30d Aug 04 12:28:52.249563 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.249533 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:52.351959 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.351920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:52.582730 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.582648 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cff8cdbd6-xbfft"] Aug 04 12:28:52.599786 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.599755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:52.622772 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.622743 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54dd7dbf45-qt86n"] Aug 04 12:28:52.626073 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.626047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.637655 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.637634 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dd7dbf45-qt86n"] Aug 04 12:28:52.675432 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.675397 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:52.711526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hk7\" (UniqueName: \"kubernetes.io/projected/4c74b51e-2661-40cc-95d0-e2c8ade362e4-kube-api-access-z9hk7\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.711654 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-serving-cert\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.711716 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-oauth-config\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.711716 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-service-ca\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.711803 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-trusted-ca-bundle\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.711873 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-oauth-serving-cert\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.711934 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.711895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-config\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-oauth-serving-cert\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-config\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813484 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hk7\" (UniqueName: \"kubernetes.io/projected/4c74b51e-2661-40cc-95d0-e2c8ade362e4-kube-api-access-z9hk7\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813484 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-serving-cert\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813585 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-oauth-config\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813585 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-service-ca\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.813676 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.813651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-trusted-ca-bundle\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.814213 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.814082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-config\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.814213 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.814156 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-oauth-serving-cert\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.814213 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.814209 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-service-ca\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.814567 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.814549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-trusted-ca-bundle\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.816478 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.816456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-oauth-config\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.816665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.816643 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-serving-cert\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.823815 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.823791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hk7\" (UniqueName: \"kubernetes.io/projected/4c74b51e-2661-40cc-95d0-e2c8ade362e4-kube-api-access-z9hk7\") pod \"console-54dd7dbf45-qt86n\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:52.935706 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:52.935670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:28:53.027265 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.027228 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" event={"ID":"fb3b212a-694c-4fd0-959c-ae6b3d3797f4","Type":"ContainerStarted","Data":"c36b96f08427f36a46494cd775ea6aeb428662fe8e2ee523e6672f2a8a1ebfd7"} Aug 04 12:28:53.027695 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.027637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:53.029526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.029499 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" exitCode=0 Aug 04 12:28:53.029655 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.029576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} Aug 04 12:28:53.029655 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.029624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"c8ccb0402ff1a2478db658241944d4de401b1251283a755253c78898c1e0e30d"} Aug 04 12:28:53.034298 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.034248 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" Aug 04 12:28:53.035846 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.035819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d"} Aug 04 12:28:53.035944 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.035892 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca"} Aug 04 12:28:53.035986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.035943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206"} Aug 04 12:28:53.036031 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.035986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerStarted","Data":"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7"} Aug 04 12:28:53.040498 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.040471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" event={"ID":"f35733d2-ce07-4b1d-ab43-bd6e9b437271","Type":"ContainerStarted","Data":"d64530a5357035546329f0fa3330bbb113387bd66d34b5806d59c5531e37a7b9"} Aug 04 12:28:53.047771 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.047046 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-2l5jz" podStartSLOduration=2.264471844 podStartE2EDuration="4.04703031s" totalImagesPullingTime="1.782558466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:53.046673346 +0000 UTC m=+64.002269219" watchObservedRunningTime="2026-08-04 12:28:53.04703031 +0000 UTC m=+64.002626108" Aug 04 12:28:53.073847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.073817 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dd7dbf45-qt86n"] Aug 04 12:28:53.076560 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:53.076539 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c74b51e_2661_40cc_95d0_e2c8ade362e4.slice/crio-22098dabb7e276e76bb805df238d16ed184dc2be59adeedc43ec92fa19d39f34 WatchSource:0}: Error finding container 22098dabb7e276e76bb805df238d16ed184dc2be59adeedc43ec92fa19d39f34: Status 404 returned error can't find the container with id 22098dabb7e276e76bb805df238d16ed184dc2be59adeedc43ec92fa19d39f34 Aug 04 12:28:53.108308 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.108261 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" podStartSLOduration=3.535678597 podStartE2EDuration="7.108243294s" totalImagesPullingTime="3.572564697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:53.106310482 +0000 UTC m=+64.061906275" watchObservedRunningTime="2026-08-04 12:28:53.108243294 +0000 UTC m=+64.063839087" Aug 04 12:28:53.161357 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.161304 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.499832398 podStartE2EDuration="8.161284752s" totalImagesPullingTime="3.640696994s" totalInitContainerRuntime="20.75536ms" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:53.157905758 +0000 UTC m=+64.113501552" watchObservedRunningTime="2026-08-04 12:28:53.161284752 +0000 UTC m=+64.116880524" Aug 04 12:28:53.918836 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.918801 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:53.983422 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:53.983383 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:54.046916 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:54.046876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dd7dbf45-qt86n" event={"ID":"4c74b51e-2661-40cc-95d0-e2c8ade362e4","Type":"ContainerStarted","Data":"e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e"} Aug 04 12:28:54.047320 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:54.046939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dd7dbf45-qt86n" event={"ID":"4c74b51e-2661-40cc-95d0-e2c8ade362e4","Type":"ContainerStarted","Data":"22098dabb7e276e76bb805df238d16ed184dc2be59adeedc43ec92fa19d39f34"} Aug 04 12:28:54.047785 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:54.047765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:54.053399 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:54.053378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-584c6647fc-tdbpc" Aug 04 12:28:54.068386 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:54.068340 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54dd7dbf45-qt86n" podStartSLOduration=2.068324201 podStartE2EDuration="2.068324201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:54.066828821 +0000 UTC m=+65.022424616" watchObservedRunningTime="2026-08-04 12:28:54.068324201 +0000 UTC m=+65.023920069" Aug 04 12:28:55.284988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.284950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:55.285447 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.285083 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:55.291573 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.291546 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:55.443790 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.443742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:55.444186 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.443943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:55.444186 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.444047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:55.446758 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.446726 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 12:28:55.446894 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.446872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 12:28:55.447016 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.446872 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 12:28:55.459103 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.457152 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 12:28:55.459103 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.457246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f3b6b1-86db-4fb6-9afd-e305159d16a2-original-pull-secret\") pod \"global-pull-secret-syncer-kd7rr\" (UID: \"c2f3b6b1-86db-4fb6-9afd-e305159d16a2\") " pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:55.459103 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.457316 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fc0096c-c114-448c-ac13-639b117d40f3-metrics-certs\") pod \"network-metrics-daemon-z28dh\" (UID: \"4fc0096c-c114-448c-ac13-639b117d40f3\") " pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:55.469283 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.469257 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlw9\" (UniqueName: \"kubernetes.io/projected/e7a58fbf-6eff-4c99-8373-7cc20e14b667-kube-api-access-wwlw9\") pod \"network-check-target-46dcw\" (UID: \"e7a58fbf-6eff-4c99-8373-7cc20e14b667\") " pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:55.561056 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.560986 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st96r\"" Aug 04 12:28:55.565690 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.565667 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p2dbw\"" Aug 04 12:28:55.567784 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.567760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd7rr" Aug 04 12:28:55.568991 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.568931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:28:55.573974 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:55.573955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z28dh" Aug 04 12:28:56.062017 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.061989 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:28:56.276809 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.276660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:56.398861 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.397811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:56.413511 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.413464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd7rr"] Aug 04 12:28:56.425732 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.425708 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z28dh"] Aug 04 12:28:56.443107 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.443079 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-46dcw"] Aug 04 12:28:56.535677 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.535643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:56.564429 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:56.564380 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f3b6b1_86db_4fb6_9afd_e305159d16a2.slice/crio-9be35a39cda65cba4c1c3a95bd86ea6ec6b989093c5b42d6643c4dfd882dc136 WatchSource:0}: Error finding container 9be35a39cda65cba4c1c3a95bd86ea6ec6b989093c5b42d6643c4dfd882dc136: Status 404 returned error can't find the container with id 9be35a39cda65cba4c1c3a95bd86ea6ec6b989093c5b42d6643c4dfd882dc136 Aug 04 12:28:56.565647 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:56.565617 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc0096c_c114_448c_ac13_639b117d40f3.slice/crio-505396c6ba6931181c174602cf53c59d9e1edbdecc2292a774bbae801aad16c4 WatchSource:0}: Error finding container 505396c6ba6931181c174602cf53c59d9e1edbdecc2292a774bbae801aad16c4: Status 404 returned error can't find the container with id 505396c6ba6931181c174602cf53c59d9e1edbdecc2292a774bbae801aad16c4 Aug 04 12:28:56.566470 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:28:56.566325 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a58fbf_6eff_4c99_8373_7cc20e14b667.slice/crio-d893ea5be43d31b4da8d766e0047deae00552ed193722ba5c6db569640a04efd WatchSource:0}: Error finding container d893ea5be43d31b4da8d766e0047deae00552ed193722ba5c6db569640a04efd: Status 404 returned error can't find the container with id d893ea5be43d31b4da8d766e0047deae00552ed193722ba5c6db569640a04efd Aug 04 12:28:56.593911 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:56.593794 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:57.064772 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.064740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-46dcw" event={"ID":"e7a58fbf-6eff-4c99-8373-7cc20e14b667","Type":"ContainerStarted","Data":"d893ea5be43d31b4da8d766e0047deae00552ed193722ba5c6db569640a04efd"} Aug 04 12:28:57.069856 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.069830 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} Aug 04 12:28:57.070035 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.069907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} Aug 04 12:28:57.070035 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.069951 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} Aug 04 12:28:57.070035 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.070023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} Aug 04 12:28:57.071967 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.071891 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z28dh" event={"ID":"4fc0096c-c114-448c-ac13-639b117d40f3","Type":"ContainerStarted","Data":"505396c6ba6931181c174602cf53c59d9e1edbdecc2292a774bbae801aad16c4"} Aug 04 12:28:57.073313 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:57.073277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd7rr" event={"ID":"c2f3b6b1-86db-4fb6-9afd-e305159d16a2","Type":"ContainerStarted","Data":"9be35a39cda65cba4c1c3a95bd86ea6ec6b989093c5b42d6643c4dfd882dc136"} Aug 04 12:28:58.085958 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.085028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} Aug 04 12:28:58.085958 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.085143 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerStarted","Data":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} Aug 04 12:28:58.088364 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.088315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z28dh" event={"ID":"4fc0096c-c114-448c-ac13-639b117d40f3","Type":"ContainerStarted","Data":"616347e1bf724f875d8346ced620060cc3ad7fde0a74690e5ab905074eabc2e2"} Aug 04 12:28:58.088364 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.088363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z28dh" event={"ID":"4fc0096c-c114-448c-ac13-639b117d40f3","Type":"ContainerStarted","Data":"6417b8269e6df5b9ad4e2c28e5885d885e8ba1fc09331a05a99ecee72e2bdc64"} Aug 04 12:28:58.125224 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.124881 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.547554593 podStartE2EDuration="7.12486422s" totalImagesPullingTime="3.558443382s" totalInitContainerRuntime="18.866245ms" isStatefulPod=false podCreationTimestamp="2026-08-04 12:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:58.123672144 +0000 UTC m=+69.079267940" watchObservedRunningTime="2026-08-04 12:28:58.12486422 +0000 UTC m=+69.080460013" Aug 04 12:28:58.143446 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.143389 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z28dh" podStartSLOduration=67.900037356 podStartE2EDuration="1m9.143372736s" totalImagesPullingTime="1.24333538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:28:58.142325213 +0000 UTC m=+69.097921010" watchObservedRunningTime="2026-08-04 12:28:58.143372736 +0000 UTC m=+69.098968530" Aug 04 12:28:58.254903 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.254863 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:58.321787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:58.321742 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:59.155311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:59.155278 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:28:59.255476 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:28:59.255436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:01.155188 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:01.155153 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:01.257187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:01.257147 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:01.734363 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:01.734272 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:29:01.988499 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:01.988419 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:02.085876 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.085843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:02.112068 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.112036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd7rr" event={"ID":"c2f3b6b1-86db-4fb6-9afd-e305159d16a2","Type":"ContainerStarted","Data":"3917c1dbd93976a0bc0563c1ba3ab88f7443710bde5590180d27f43eb4be03c9"} Aug 04 12:29:02.113698 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.113673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-46dcw" event={"ID":"e7a58fbf-6eff-4c99-8373-7cc20e14b667","Type":"ContainerStarted","Data":"2a3145d57c7bfc4d2716385ca80a533672bd102c0d97f592665c2efd0c14eb36"} Aug 04 12:29:02.113827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.113721 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:29:02.149367 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.149316 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kd7rr" podStartSLOduration=68.375143435 podStartE2EDuration="1m13.14929932s" totalImagesPullingTime="4.774155885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:29:02.146943302 +0000 UTC m=+73.102539095" watchObservedRunningTime="2026-08-04 12:29:02.14929932 +0000 UTC m=+73.104895114" Aug 04 12:29:02.212405 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.212354 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-46dcw" podStartSLOduration=68.443229474 podStartE2EDuration="1m13.212338141s" totalImagesPullingTime="4.769108667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:29:02.211066963 +0000 UTC m=+73.166662758" watchObservedRunningTime="2026-08-04 12:29:02.212338141 +0000 UTC m=+73.167933935" Aug 04 12:29:02.936441 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.936403 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:29:02.936647 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.936462 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:29:02.941383 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:02.941357 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:29:03.003450 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:03.003408 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:03.098347 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:03.098313 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:03.122273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:03.122249 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:29:03.200589 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:03.200497 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbbb79648-kpdx7"] Aug 04 12:29:03.572453 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:03.572367 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:05.388422 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:05.388386 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:05.492007 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:05.491971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:07.216609 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:07.216579 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:07.319551 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:07.319522 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:09.639976 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:09.639927 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:29:09.640433 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:09.640027 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:29:17.602680 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.602617 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cff8cdbd6-xbfft" podUID="d2133776-e891-410f-9988-a0e839e0cc21" containerName="console" containerID="cri-o://f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15" gracePeriod=15 Aug 04 12:29:17.844829 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.844802 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:29:17.954434 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-oauth-config\") pod \"d2133776-e891-410f-9988-a0e839e0cc21\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " Aug 04 12:29:17.954584 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954446 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-oauth-serving-cert\") pod \"d2133776-e891-410f-9988-a0e839e0cc21\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " Aug 04 12:29:17.954584 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954493 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwcmw\" (UniqueName: \"kubernetes.io/projected/d2133776-e891-410f-9988-a0e839e0cc21-kube-api-access-qwcmw\") pod \"d2133776-e891-410f-9988-a0e839e0cc21\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " Aug 04 12:29:17.954722 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954694 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-service-ca\") pod \"d2133776-e891-410f-9988-a0e839e0cc21\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " Aug 04 12:29:17.954792 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954751 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-console-config\") pod \"d2133776-e891-410f-9988-a0e839e0cc21\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " Aug 04 12:29:17.954843 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954800 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-serving-cert\") pod \"d2133776-e891-410f-9988-a0e839e0cc21\" (UID: \"d2133776-e891-410f-9988-a0e839e0cc21\") " Aug 04 12:29:17.954843 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.954807 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d2133776-e891-410f-9988-a0e839e0cc21" (UID: "d2133776-e891-410f-9988-a0e839e0cc21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:17.955076 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.955036 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-service-ca" (OuterVolumeSpecName: "service-ca") pod "d2133776-e891-410f-9988-a0e839e0cc21" (UID: "d2133776-e891-410f-9988-a0e839e0cc21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:17.955212 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.955078 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-console-config" (OuterVolumeSpecName: "console-config") pod "d2133776-e891-410f-9988-a0e839e0cc21" (UID: "d2133776-e891-410f-9988-a0e839e0cc21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:17.955212 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.955105 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-oauth-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:17.957365 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.957339 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d2133776-e891-410f-9988-a0e839e0cc21" (UID: "d2133776-e891-410f-9988-a0e839e0cc21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:29:17.957467 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.957351 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2133776-e891-410f-9988-a0e839e0cc21-kube-api-access-qwcmw" (OuterVolumeSpecName: "kube-api-access-qwcmw") pod "d2133776-e891-410f-9988-a0e839e0cc21" (UID: "d2133776-e891-410f-9988-a0e839e0cc21"). InnerVolumeSpecName "kube-api-access-qwcmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:29:17.957467 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:17.957400 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d2133776-e891-410f-9988-a0e839e0cc21" (UID: "d2133776-e891-410f-9988-a0e839e0cc21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:29:18.055954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.055904 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-oauth-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:18.055954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.055940 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcmw\" (UniqueName: \"kubernetes.io/projected/d2133776-e891-410f-9988-a0e839e0cc21-kube-api-access-qwcmw\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:18.055954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.055952 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:18.055954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.055960 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2133776-e891-410f-9988-a0e839e0cc21-console-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:18.055954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.055969 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2133776-e891-410f-9988-a0e839e0cc21-console-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:18.183589 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.183558 2578 generic.go:356] "Generic (PLEG): container finished" podID="d2133776-e891-410f-9988-a0e839e0cc21" containerID="f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15" exitCode=0 Aug 04 12:29:18.183745 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.183608 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cff8cdbd6-xbfft" Aug 04 12:29:18.183745 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.183623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cff8cdbd6-xbfft" event={"ID":"d2133776-e891-410f-9988-a0e839e0cc21","Type":"ContainerDied","Data":"f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15"} Aug 04 12:29:18.183745 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.183663 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cff8cdbd6-xbfft" event={"ID":"d2133776-e891-410f-9988-a0e839e0cc21","Type":"ContainerDied","Data":"f6de99e2fe2c0d17f787c748f21511fc93241a396cae2c13be554691362e2503"} Aug 04 12:29:18.183745 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.183683 2578 scope.go:117] "RemoveContainer" containerID="f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15" Aug 04 12:29:18.192785 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.192769 2578 scope.go:117] "RemoveContainer" containerID="f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15" Aug 04 12:29:18.193041 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:29:18.193020 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15\": container with ID starting with f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15 not found: ID does not exist" containerID="f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15" Aug 04 12:29:18.193108 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.193053 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15"} err="failed to get container status \"f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15\": rpc error: code = NotFound desc = could not find container \"f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15\": container with ID starting with f8b87316cfed92eb91d7d5cf13de35b6c1a20cc9e4297baf6d1f481ad28beb15 not found: ID does not exist" Aug 04 12:29:18.209750 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.209695 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cff8cdbd6-xbfft"] Aug 04 12:29:18.214858 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.214836 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cff8cdbd6-xbfft"] Aug 04 12:29:18.790531 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:18.790497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:19.552645 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:19.552614 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2133776-e891-410f-9988-a0e839e0cc21" path="/var/lib/kubelet/pods/d2133776-e891-410f-9988-a0e839e0cc21/volumes" Aug 04 12:29:22.201030 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:22.200995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:22.301068 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:22.301034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:22.794746 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:22.794711 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:23.498187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:23.498154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:29:28.220955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.220908 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bbbb79648-kpdx7" podUID="c60e4ff1-c0e6-4981-aa47-c65141dd96a6" containerName="console" containerID="cri-o://bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3" gracePeriod=15 Aug 04 12:29:28.474634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.474578 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:29:28.541481 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-oauth-config\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.541675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-service-ca\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.541675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541536 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d5vh\" (UniqueName: \"kubernetes.io/projected/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-kube-api-access-6d5vh\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.541675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541566 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-config\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.541675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541615 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-trusted-ca-bundle\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.541675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-serving-cert\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.541932 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.541696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-oauth-serving-cert\") pod \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\" (UID: \"c60e4ff1-c0e6-4981-aa47-c65141dd96a6\") " Aug 04 12:29:28.542081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.542036 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:28.542236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.542079 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-config" (OuterVolumeSpecName: "console-config") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:28.542236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.542082 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:28.542236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.542174 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:29:28.544059 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.544034 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-kube-api-access-6d5vh" (OuterVolumeSpecName: "kube-api-access-6d5vh") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "kube-api-access-6d5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:29:28.544168 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.544033 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:29:28.544168 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.544098 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c60e4ff1-c0e6-4981-aa47-c65141dd96a6" (UID: "c60e4ff1-c0e6-4981-aa47-c65141dd96a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:29:28.642456 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642420 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:28.642456 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642450 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-trusted-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:28.642456 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642461 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:28.642687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642470 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-oauth-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:28.642687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642478 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-console-oauth-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:28.642687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642486 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:28.642687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:28.642494 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5vh\" (UniqueName: \"kubernetes.io/projected/c60e4ff1-c0e6-4981-aa47-c65141dd96a6-kube-api-access-6d5vh\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:29:29.235374 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.235343 2578 generic.go:356] "Generic (PLEG): container finished" podID="c60e4ff1-c0e6-4981-aa47-c65141dd96a6" containerID="bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3" exitCode=0 Aug 04 12:29:29.235784 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.235386 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbbb79648-kpdx7" event={"ID":"c60e4ff1-c0e6-4981-aa47-c65141dd96a6","Type":"ContainerDied","Data":"bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3"} Aug 04 12:29:29.235784 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.235411 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bbbb79648-kpdx7" Aug 04 12:29:29.235784 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.235429 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bbbb79648-kpdx7" event={"ID":"c60e4ff1-c0e6-4981-aa47-c65141dd96a6","Type":"ContainerDied","Data":"441dd197431406d8ff9ea40bcb1af7a1c8dfa92588aadf39fc98124513d67a6e"} Aug 04 12:29:29.235784 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.235447 2578 scope.go:117] "RemoveContainer" containerID="bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3" Aug 04 12:29:29.244575 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.244556 2578 scope.go:117] "RemoveContainer" containerID="bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3" Aug 04 12:29:29.244879 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:29:29.244858 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3\": container with ID starting with bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3 not found: ID does not exist" containerID="bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3" Aug 04 12:29:29.244940 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.244888 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3"} err="failed to get container status \"bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3\": rpc error: code = NotFound desc = could not find container \"bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3\": container with ID starting with bf54d99ee7b3b2a400efcdc23946f81f74204a1197d6ea74446e4d3074e0cda3 not found: ID does not exist" Aug 04 12:29:29.259326 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.259301 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bbbb79648-kpdx7"] Aug 04 12:29:29.264601 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.264575 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bbbb79648-kpdx7"] Aug 04 12:29:29.552346 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.552266 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60e4ff1-c0e6-4981-aa47-c65141dd96a6" path="/var/lib/kubelet/pods/c60e4ff1-c0e6-4981-aa47-c65141dd96a6/volumes" Aug 04 12:29:29.645662 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.645638 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:29:29.649958 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:29.649929 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d76c6449f-jp7q7" Aug 04 12:29:33.120579 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:33.120549 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-46dcw" Aug 04 12:29:51.735293 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:51.735241 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:29:51.755343 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:51.755316 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:29:52.348927 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:52.348901 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:29:59.675698 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:29:59.675659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:30:04.348876 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:04.348842 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:30:04.653300 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:04.653212 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:30:05.422011 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.421976 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:30:05.422429 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.422307 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="alertmanager" containerID="cri-o://53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987" gracePeriod=120 Aug 04 12:30:05.422429 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.422383 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-metric" containerID="cri-o://bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca" gracePeriod=120 Aug 04 12:30:05.422429 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.422404 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-web" containerID="cri-o://9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7" gracePeriod=120 Aug 04 12:30:05.422602 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.422442 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="prom-label-proxy" containerID="cri-o://2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d" gracePeriod=120 Aug 04 12:30:05.422602 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.422473 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="config-reloader" containerID="cri-o://4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447" gracePeriod=120 Aug 04 12:30:05.422602 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:05.422449 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy" containerID="cri-o://580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206" gracePeriod=120 Aug 04 12:30:06.393249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393217 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d" exitCode=0 Aug 04 12:30:06.393249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393242 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206" exitCode=0 Aug 04 12:30:06.393249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393248 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447" exitCode=0 Aug 04 12:30:06.393249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393254 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987" exitCode=0 Aug 04 12:30:06.393503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d"} Aug 04 12:30:06.393503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206"} Aug 04 12:30:06.393503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447"} Aug 04 12:30:06.393503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.393407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987"} Aug 04 12:30:06.663881 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.663859 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:06.677777 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677753 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-main-tls\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.677847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677788 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-web\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.677847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-out\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.677847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677829 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.677847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9bgf\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-kube-api-access-m9bgf\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677861 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-metrics-client-ca\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677881 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-tls-assets\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677928 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-trusted-ca-bundle\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.677969 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-cluster-tls-config\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.678013 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-web-config\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.678035 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-volume\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678359 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.678055 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.678359 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.678145 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-main-db\") pod \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\" (UID: \"2aae3095-5bb2-4bd4-8e8f-d109436b765f\") " Aug 04 12:30:06.679241 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.678726 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:06.679503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.679361 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:06.679503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.679457 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:30:06.681827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.681645 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.682369 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682337 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-out" (OuterVolumeSpecName: "config-out") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:30:06.682369 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682343 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-kube-api-access-m9bgf" (OuterVolumeSpecName: "kube-api-access-m9bgf") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "kube-api-access-m9bgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:30:06.682515 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682398 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.682570 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682535 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.682693 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682666 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.682981 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682960 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:30:06.683064 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.682992 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-volume" (OuterVolumeSpecName: "config-volume") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.686331 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.686306 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.694617 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.694593 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-web-config" (OuterVolumeSpecName: "web-config") pod "2aae3095-5bb2-4bd4-8e8f-d109436b765f" (UID: "2aae3095-5bb2-4bd4-8e8f-d109436b765f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:06.779485 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779453 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779485 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779483 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-cluster-tls-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779513 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-web-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779522 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779531 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779539 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-alertmanager-main-db\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779548 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-main-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779557 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779569 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aae3095-5bb2-4bd4-8e8f-d109436b765f-config-out\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779578 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2aae3095-5bb2-4bd4-8e8f-d109436b765f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779587 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bgf\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-kube-api-access-m9bgf\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779597 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aae3095-5bb2-4bd4-8e8f-d109436b765f-metrics-client-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:06.779675 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:06.779605 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aae3095-5bb2-4bd4-8e8f-d109436b765f-tls-assets\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:07.400760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400726 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca" exitCode=0 Aug 04 12:30:07.400760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400756 2578 generic.go:356] "Generic (PLEG): container finished" podID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerID="9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7" exitCode=0 Aug 04 12:30:07.401020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400817 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.401020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca"} Aug 04 12:30:07.401020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7"} Aug 04 12:30:07.401020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2aae3095-5bb2-4bd4-8e8f-d109436b765f","Type":"ContainerDied","Data":"2d50f7903da26a0cc7e2b71dc5527556a240a73ef70f2b5d850accd4cf2bf5ce"} Aug 04 12:30:07.401020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.400973 2578 scope.go:117] "RemoveContainer" containerID="2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d" Aug 04 12:30:07.409246 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.409203 2578 scope.go:117] "RemoveContainer" containerID="bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca" Aug 04 12:30:07.416976 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.416957 2578 scope.go:117] "RemoveContainer" containerID="580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206" Aug 04 12:30:07.423920 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.423905 2578 scope.go:117] "RemoveContainer" containerID="9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7" Aug 04 12:30:07.427746 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.427721 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:30:07.431748 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.431730 2578 scope.go:117] "RemoveContainer" containerID="4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447" Aug 04 12:30:07.435726 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.435702 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:30:07.440810 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.440792 2578 scope.go:117] "RemoveContainer" containerID="53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987" Aug 04 12:30:07.448847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.448831 2578 scope.go:117] "RemoveContainer" containerID="a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7" Aug 04 12:30:07.456174 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.456116 2578 scope.go:117] "RemoveContainer" containerID="2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d" Aug 04 12:30:07.456453 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.456431 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d\": container with ID starting with 2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d not found: ID does not exist" containerID="2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d" Aug 04 12:30:07.456511 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.456463 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d"} err="failed to get container status \"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d\": rpc error: code = NotFound desc = could not find container \"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d\": container with ID starting with 2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d not found: ID does not exist" Aug 04 12:30:07.456511 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.456483 2578 scope.go:117] "RemoveContainer" containerID="bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca" Aug 04 12:30:07.456771 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.456746 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca\": container with ID starting with bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca not found: ID does not exist" containerID="bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca" Aug 04 12:30:07.456812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.456778 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca"} err="failed to get container status \"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca\": rpc error: code = NotFound desc = could not find container \"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca\": container with ID starting with bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca not found: ID does not exist" Aug 04 12:30:07.456812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.456795 2578 scope.go:117] "RemoveContainer" containerID="580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206" Aug 04 12:30:07.457066 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.457047 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206\": container with ID starting with 580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206 not found: ID does not exist" containerID="580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206" Aug 04 12:30:07.457106 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457074 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206"} err="failed to get container status \"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206\": rpc error: code = NotFound desc = could not find container \"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206\": container with ID starting with 580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206 not found: ID does not exist" Aug 04 12:30:07.457106 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457095 2578 scope.go:117] "RemoveContainer" containerID="9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7" Aug 04 12:30:07.457367 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.457345 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7\": container with ID starting with 9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7 not found: ID does not exist" containerID="9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7" Aug 04 12:30:07.457413 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457370 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7"} err="failed to get container status \"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7\": rpc error: code = NotFound desc = could not find container \"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7\": container with ID starting with 9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7 not found: ID does not exist" Aug 04 12:30:07.457413 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457383 2578 scope.go:117] "RemoveContainer" containerID="4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447" Aug 04 12:30:07.457628 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.457610 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447\": container with ID starting with 4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447 not found: ID does not exist" containerID="4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447" Aug 04 12:30:07.457668 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457634 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447"} err="failed to get container status \"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447\": rpc error: code = NotFound desc = could not find container \"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447\": container with ID starting with 4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447 not found: ID does not exist" Aug 04 12:30:07.457668 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457650 2578 scope.go:117] "RemoveContainer" containerID="53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987" Aug 04 12:30:07.457854 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.457838 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987\": container with ID starting with 53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987 not found: ID does not exist" containerID="53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987" Aug 04 12:30:07.457899 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457856 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987"} err="failed to get container status \"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987\": rpc error: code = NotFound desc = could not find container \"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987\": container with ID starting with 53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987 not found: ID does not exist" Aug 04 12:30:07.457899 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.457868 2578 scope.go:117] "RemoveContainer" containerID="a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7" Aug 04 12:30:07.458100 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:07.458083 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7\": container with ID starting with a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7 not found: ID does not exist" containerID="a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7" Aug 04 12:30:07.458171 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458141 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7"} err="failed to get container status \"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7\": rpc error: code = NotFound desc = could not find container \"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7\": container with ID starting with a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7 not found: ID does not exist" Aug 04 12:30:07.458171 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458158 2578 scope.go:117] "RemoveContainer" containerID="2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d" Aug 04 12:30:07.458398 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458380 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d"} err="failed to get container status \"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d\": rpc error: code = NotFound desc = could not find container \"2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d\": container with ID starting with 2d8b0197d71dc06f4919fa506379a206ae19d34925da9f41d9badaa4509e8e7d not found: ID does not exist" Aug 04 12:30:07.458459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458399 2578 scope.go:117] "RemoveContainer" containerID="bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca" Aug 04 12:30:07.458627 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458605 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca"} err="failed to get container status \"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca\": rpc error: code = NotFound desc = could not find container \"bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca\": container with ID starting with bcb157c0ad333bb475beda497384f2b13bf722e439a1b23f45f9b91a261321ca not found: ID does not exist" Aug 04 12:30:07.458627 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458625 2578 scope.go:117] "RemoveContainer" containerID="580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206" Aug 04 12:30:07.458829 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458814 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206"} err="failed to get container status \"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206\": rpc error: code = NotFound desc = could not find container \"580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206\": container with ID starting with 580c16e8a6b38ab5c8dcb87c59f712ab7438f1d575e7b4173eebb2d811a2f206 not found: ID does not exist" Aug 04 12:30:07.458829 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.458829 2578 scope.go:117] "RemoveContainer" containerID="9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7" Aug 04 12:30:07.459023 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459009 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7"} err="failed to get container status \"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7\": rpc error: code = NotFound desc = could not find container \"9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7\": container with ID starting with 9411227cc28c44d7e961daa2c5399504b31b83166f459551e3f9b314c388d2d7 not found: ID does not exist" Aug 04 12:30:07.459073 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459023 2578 scope.go:117] "RemoveContainer" containerID="4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447" Aug 04 12:30:07.459266 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459239 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447"} err="failed to get container status \"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447\": rpc error: code = NotFound desc = could not find container \"4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447\": container with ID starting with 4925e80a2698c80ffae2c75a2a280c136d2dc18a52088d4612905c61152e4447 not found: ID does not exist" Aug 04 12:30:07.459357 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459266 2578 scope.go:117] "RemoveContainer" containerID="53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987" Aug 04 12:30:07.459547 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459523 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987"} err="failed to get container status \"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987\": rpc error: code = NotFound desc = could not find container \"53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987\": container with ID starting with 53f0294f889e34a743e3e11c0d6b03a3996420519aaecd646680fa441b4b4987 not found: ID does not exist" Aug 04 12:30:07.459547 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459547 2578 scope.go:117] "RemoveContainer" containerID="a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7" Aug 04 12:30:07.459895 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.459867 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7"} err="failed to get container status \"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7\": rpc error: code = NotFound desc = could not find container \"a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7\": container with ID starting with a14bab935d8ba9c327a5869a2403c2dfc864ba4edbce715f5223c30c4b97d0d7 not found: ID does not exist" Aug 04 12:30:07.470011 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.469982 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:30:07.470324 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470309 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy" Aug 04 12:30:07.470324 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470324 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470335 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="config-reloader" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470341 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="config-reloader" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470348 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-web" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470354 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-web" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470365 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2133776-e891-410f-9988-a0e839e0cc21" containerName="console" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470373 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2133776-e891-410f-9988-a0e839e0cc21" containerName="console" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470380 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-metric" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470386 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-metric" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470392 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="prom-label-proxy" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470396 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="prom-label-proxy" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470405 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60e4ff1-c0e6-4981-aa47-c65141dd96a6" containerName="console" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470410 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60e4ff1-c0e6-4981-aa47-c65141dd96a6" containerName="console" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470416 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="alertmanager" Aug 04 12:30:07.470420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470421 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="alertmanager" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470431 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="init-config-reloader" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470438 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="init-config-reloader" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470499 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2133776-e891-410f-9988-a0e839e0cc21" containerName="console" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470510 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60e4ff1-c0e6-4981-aa47-c65141dd96a6" containerName="console" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470518 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470529 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="config-reloader" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470537 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="prom-label-proxy" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470547 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-web" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470555 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="alertmanager" Aug 04 12:30:07.470968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.470562 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" containerName="kube-rbac-proxy-metric" Aug 04 12:30:07.477039 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.477017 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.480359 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480340 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 12:30:07.480472 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480343 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 12:30:07.480619 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480600 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 12:30:07.480694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480635 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z7xzm\"" Aug 04 12:30:07.480694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480600 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 12:30:07.480694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480597 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 12:30:07.480844 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480608 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 12:30:07.480912 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480898 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 12:30:07.480971 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.480933 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 12:30:07.486182 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54742e38-f7aa-41b1-8181-14dcb22d11bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486287 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486287 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486287 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9q25\" (UniqueName: \"kubernetes.io/projected/54742e38-f7aa-41b1-8181-14dcb22d11bb-kube-api-access-f9q25\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54742e38-f7aa-41b1-8181-14dcb22d11bb-config-out\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-web-config\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54742e38-f7aa-41b1-8181-14dcb22d11bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54742e38-f7aa-41b1-8181-14dcb22d11bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.486708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.486606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54742e38-f7aa-41b1-8181-14dcb22d11bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.490967 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.489552 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 12:30:07.492165 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.492140 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:30:07.551757 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.551721 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aae3095-5bb2-4bd4-8e8f-d109436b765f" path="/var/lib/kubelet/pods/2aae3095-5bb2-4bd4-8e8f-d109436b765f/volumes" Aug 04 12:30:07.587555 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9q25\" (UniqueName: \"kubernetes.io/projected/54742e38-f7aa-41b1-8181-14dcb22d11bb-kube-api-access-f9q25\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.587555 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.587802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54742e38-f7aa-41b1-8181-14dcb22d11bb-config-out\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.587802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.587802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-web-config\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.587802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54742e38-f7aa-41b1-8181-14dcb22d11bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.587802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54742e38-f7aa-41b1-8181-14dcb22d11bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54742e38-f7aa-41b1-8181-14dcb22d11bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54742e38-f7aa-41b1-8181-14dcb22d11bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.587986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.588655 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.588462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54742e38-f7aa-41b1-8181-14dcb22d11bb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.589039 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.588991 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54742e38-f7aa-41b1-8181-14dcb22d11bb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.589945 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.589922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54742e38-f7aa-41b1-8181-14dcb22d11bb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.590807 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.590755 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54742e38-f7aa-41b1-8181-14dcb22d11bb-config-out\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.591063 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.591038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-config-volume\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.591185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.591160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-web-config\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.591376 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.591351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.591457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.591439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.592303 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.592274 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.592486 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.592468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54742e38-f7aa-41b1-8181-14dcb22d11bb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.592792 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.592769 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.592964 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.592943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54742e38-f7aa-41b1-8181-14dcb22d11bb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.597955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.597933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9q25\" (UniqueName: \"kubernetes.io/projected/54742e38-f7aa-41b1-8181-14dcb22d11bb-kube-api-access-f9q25\") pod \"alertmanager-main-0\" (UID: \"54742e38-f7aa-41b1-8181-14dcb22d11bb\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.790093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.790015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 12:30:07.955001 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:07.954911 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 12:30:07.957209 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:30:07.957175 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54742e38_f7aa_41b1_8181_14dcb22d11bb.slice/crio-aa4a6b2ef0b919397a042ad840b7d74f2a2ee9cde4761886e67fc2a50188a335 WatchSource:0}: Error finding container aa4a6b2ef0b919397a042ad840b7d74f2a2ee9cde4761886e67fc2a50188a335: Status 404 returned error can't find the container with id aa4a6b2ef0b919397a042ad840b7d74f2a2ee9cde4761886e67fc2a50188a335 Aug 04 12:30:08.406700 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:08.406664 2578 generic.go:356] "Generic (PLEG): container finished" podID="54742e38-f7aa-41b1-8181-14dcb22d11bb" containerID="7a0cbcf87193ed43c3928ddd76d0f1a171e73309bd315be12eca8b4469b1a875" exitCode=0 Aug 04 12:30:08.406875 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:08.406715 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerDied","Data":"7a0cbcf87193ed43c3928ddd76d0f1a171e73309bd315be12eca8b4469b1a875"} Aug 04 12:30:08.406875 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:08.406782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"aa4a6b2ef0b919397a042ad840b7d74f2a2ee9cde4761886e67fc2a50188a335"} Aug 04 12:30:09.422238 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.422196 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"a484145f610ca3ebfa3757a84d2a2628350712ab1330953dce55696baed11330"} Aug 04 12:30:09.422640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.422330 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"938ece349ccae96500d415f0c12f6c5321ede8938ee14a954e45d0956a5ce0a7"} Aug 04 12:30:09.422640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.422380 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"b770f75949553ec83d1ee4fcf8277bc9189bcc9333cbaad7c77fadfe778fb27e"} Aug 04 12:30:09.422640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.422430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"16b640a8c09b3a5cd57ab27b6d74e1be43626945c00658805c17526bd1b96cad"} Aug 04 12:30:09.422640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.422471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"ca6d2b4c28190c14e128c398d88e818b8f1d9147b092793dd0fe434c9da934ba"} Aug 04 12:30:09.422640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.422513 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54742e38-f7aa-41b1-8181-14dcb22d11bb","Type":"ContainerStarted","Data":"e41710a60bc10a848e0f0f2c2fab34d6d2671950b7332da79ec91654c65191a3"} Aug 04 12:30:09.464547 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.464486 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.449442852 podStartE2EDuration="2.46446416s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.021308ms" isStatefulPod=false podCreationTimestamp="2026-08-04 12:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:30:09.462905571 +0000 UTC m=+140.418501366" watchObservedRunningTime="2026-08-04 12:30:09.46446416 +0000 UTC m=+140.420059956" Aug 04 12:30:09.827443 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.827352 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:30:09.828003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.827956 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="prometheus" containerID="cri-o://e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" gracePeriod=600 Aug 04 12:30:09.828166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.827989 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" gracePeriod=600 Aug 04 12:30:09.828166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.828052 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-web" containerID="cri-o://47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" gracePeriod=600 Aug 04 12:30:09.828166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.828077 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy" containerID="cri-o://e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" gracePeriod=600 Aug 04 12:30:09.828166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.828089 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="config-reloader" containerID="cri-o://a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" gracePeriod=600 Aug 04 12:30:09.828379 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:09.828228 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="thanos-sidecar" containerID="cri-o://d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" gracePeriod=600 Aug 04 12:30:10.081847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.081786 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.111208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111163 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwdrx\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-kube-api-access-pwdrx\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-metrics-client-ca\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111425 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111237 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-0\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111480 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111434 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111549 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111491 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-kubelet-serving-ca-bundle\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111549 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111521 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-tls\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-1\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111581 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.111665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111593 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-db\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111616 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-2\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111665 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111643 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-trusted-ca-bundle\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111719 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-tls-assets\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111747 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-serving-certs-ca-bundle\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111783 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-web-config\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111809 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-thanos-prometheus-http-client-file\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111825 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-grpc-tls\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111877 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config-out\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111902 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.111966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111934 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.112471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.111974 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-kube-rbac-proxy\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.112471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.112010 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-metrics-client-certs\") pod \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\" (UID: \"b6e4fe5e-5491-4c35-b219-82cc3a217ca8\") " Aug 04 12:30:10.112471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.112310 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-metrics-client-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.112471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.112331 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.112471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.112346 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.113018 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.112790 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.114202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.113594 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.114202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.113835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.115857 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.115829 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:30:10.116232 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.116171 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:10.119780 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.116383 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.120170 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.119983 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.120757 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.120351 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-kube-api-access-pwdrx" (OuterVolumeSpecName: "kube-api-access-pwdrx") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "kube-api-access-pwdrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:30:10.120757 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.120475 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:30:10.120757 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.120550 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.120757 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.120695 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.121276 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.121237 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.121382 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.121315 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.121521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.121487 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config-out" (OuterVolumeSpecName: "config-out") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:30:10.121593 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.121568 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config" (OuterVolumeSpecName: "config") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.121788 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.121761 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.134325 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.134286 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-web-config" (OuterVolumeSpecName: "web-config") pod "b6e4fe5e-5491-4c35-b219-82cc3a217ca8" (UID: "b6e4fe5e-5491-4c35-b219-82cc3a217ca8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:10.213630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213596 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213624 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213635 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213645 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-db\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213656 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213664 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213673 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-tls-assets\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213681 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213689 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-web-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213697 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213704 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-grpc-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213713 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config-out\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213721 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213728 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-kube-rbac-proxy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213735 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-secret-metrics-client-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213744 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdrx\" (UniqueName: \"kubernetes.io/projected/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-kube-api-access-pwdrx\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.213847 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.213751 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6e4fe5e-5491-4c35-b219-82cc3a217ca8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:10.430391 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430357 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" exitCode=0 Aug 04 12:30:10.430391 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430387 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" exitCode=0 Aug 04 12:30:10.430391 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430396 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" exitCode=0 Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430404 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" exitCode=0 Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430412 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" exitCode=0 Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430420 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" exitCode=0 Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430452 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430573 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} Aug 04 12:30:10.430862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.430632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6e4fe5e-5491-4c35-b219-82cc3a217ca8","Type":"ContainerDied","Data":"c8ccb0402ff1a2478db658241944d4de401b1251283a755253c78898c1e0e30d"} Aug 04 12:30:10.438565 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.438507 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.445684 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.445667 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.452447 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.452432 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.458545 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.458522 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:30:10.460042 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.460024 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.472319 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.472292 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:30:10.472818 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.472798 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.491222 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.491198 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.497578 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497554 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:30:10.497918 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497903 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497921 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497933 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-thanos" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497941 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-thanos" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497957 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-web" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497965 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-web" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497986 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="config-reloader" Aug 04 12:30:10.498003 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.497995 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="config-reloader" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498008 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="prometheus" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498016 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="prometheus" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498033 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="init-config-reloader" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498041 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="init-config-reloader" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498061 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="thanos-sidecar" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498069 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="thanos-sidecar" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498172 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-thanos" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498185 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="prometheus" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498194 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy-web" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498203 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="config-reloader" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498213 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="thanos-sidecar" Aug 04 12:30:10.498355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498224 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" containerName="kube-rbac-proxy" Aug 04 12:30:10.498891 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.498827 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.499166 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.499145 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.499239 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499175 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} err="failed to get container status \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" Aug 04 12:30:10.499239 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499196 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.499457 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.499440 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.499510 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499462 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} err="failed to get container status \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" Aug 04 12:30:10.499510 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499475 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.499703 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.499682 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.499760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499711 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} err="failed to get container status \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" Aug 04 12:30:10.499760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499727 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.499938 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.499920 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.499982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499940 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} err="failed to get container status \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" Aug 04 12:30:10.499982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.499952 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.500182 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.500159 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.500224 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.500192 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} err="failed to get container status \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" Aug 04 12:30:10.500224 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.500214 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.500487 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.500466 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.500581 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.500489 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} err="failed to get container status \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" Aug 04 12:30:10.500581 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.500504 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.500761 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:10.500744 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.500799 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.500765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} err="failed to get container status \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" Aug 04 12:30:10.500799 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.500779 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.501195 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501082 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} err="failed to get container status \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" Aug 04 12:30:10.501195 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501108 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.501426 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501404 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} err="failed to get container status \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" Aug 04 12:30:10.501508 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501428 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.501708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501690 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} err="failed to get container status \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" Aug 04 12:30:10.501750 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501709 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.501953 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501937 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} err="failed to get container status \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" Aug 04 12:30:10.501953 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.501953 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.502271 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.502249 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} err="failed to get container status \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" Aug 04 12:30:10.502271 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.502272 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.502560 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.502540 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} err="failed to get container status \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" Aug 04 12:30:10.502560 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.502559 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.502811 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.502790 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} err="failed to get container status \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" Aug 04 12:30:10.502811 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.502811 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.503025 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503008 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} err="failed to get container status \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" Aug 04 12:30:10.503086 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503025 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.503247 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503229 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} err="failed to get container status \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" Aug 04 12:30:10.503247 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503247 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.503500 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503483 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} err="failed to get container status \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" Aug 04 12:30:10.503544 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503501 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.503721 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503695 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} err="failed to get container status \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" Aug 04 12:30:10.503765 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503723 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.503904 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503879 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} err="failed to get container status \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" Aug 04 12:30:10.503980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.503905 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.504188 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504171 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} err="failed to get container status \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" Aug 04 12:30:10.504241 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504189 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.504530 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504512 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} err="failed to get container status \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" Aug 04 12:30:10.504572 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504531 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.504760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504742 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} err="failed to get container status \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" Aug 04 12:30:10.504837 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504762 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.504938 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.505021 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.504994 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} err="failed to get container status \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" Aug 04 12:30:10.505021 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.505020 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.505557 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.505524 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} err="failed to get container status \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" Aug 04 12:30:10.505642 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.505560 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.505849 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.505823 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} err="failed to get container status \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" Aug 04 12:30:10.505905 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.505850 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.506051 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506026 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} err="failed to get container status \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" Aug 04 12:30:10.506051 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506043 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.506399 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506380 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} err="failed to get container status \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" Aug 04 12:30:10.506451 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506402 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.506634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506617 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} err="failed to get container status \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" Aug 04 12:30:10.506678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506635 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.506924 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506902 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} err="failed to get container status \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" Aug 04 12:30:10.506966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.506929 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.507212 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507192 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} err="failed to get container status \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" Aug 04 12:30:10.507296 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507214 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.507432 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507414 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} err="failed to get container status \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" Aug 04 12:30:10.507472 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507433 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.507670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507652 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} err="failed to get container status \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" Aug 04 12:30:10.507735 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507672 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.507915 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507892 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} err="failed to get container status \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" Aug 04 12:30:10.507915 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.507913 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.508207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508185 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} err="failed to get container status \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" Aug 04 12:30:10.508207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508207 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.508462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508444 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} err="failed to get container status \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" Aug 04 12:30:10.508462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508461 2578 scope.go:117] "RemoveContainer" containerID="cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1" Aug 04 12:30:10.508692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508669 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1"} err="failed to get container status \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": rpc error: code = NotFound desc = could not find container \"cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1\": container with ID starting with cb74ab746f92027faf2267ac884355c3920ee793f2ae155bde6becaf4151e0b1 not found: ID does not exist" Aug 04 12:30:10.508752 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508692 2578 scope.go:117] "RemoveContainer" containerID="e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae" Aug 04 12:30:10.508917 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508899 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae"} err="failed to get container status \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": rpc error: code = NotFound desc = could not find container \"e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae\": container with ID starting with e507f8391e6d3c28fb4d7f00ccabbcabf276fc721b5b5fa6448a8943995734ae not found: ID does not exist" Aug 04 12:30:10.508962 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.508919 2578 scope.go:117] "RemoveContainer" containerID="47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86" Aug 04 12:30:10.509177 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509154 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86"} err="failed to get container status \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": rpc error: code = NotFound desc = could not find container \"47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86\": container with ID starting with 47dc4aff624f97c7e49f61968adf682bbc5ee2738e24e437a780b4e6673a9b86 not found: ID does not exist" Aug 04 12:30:10.509272 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509178 2578 scope.go:117] "RemoveContainer" containerID="d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de" Aug 04 12:30:10.509595 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509576 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de"} err="failed to get container status \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": rpc error: code = NotFound desc = could not find container \"d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de\": container with ID starting with d670295a8bb2d160b509a07d38bcaaa35bd5a900e675f744d2eac3be77f686de not found: ID does not exist" Aug 04 12:30:10.509670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509597 2578 scope.go:117] "RemoveContainer" containerID="a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737" Aug 04 12:30:10.509713 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 04 12:30:10.509713 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509688 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 04 12:30:10.509927 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509911 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 04 12:30:10.510058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509955 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737"} err="failed to get container status \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": rpc error: code = NotFound desc = could not find container \"a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737\": container with ID starting with a4fed43be699e38751518fdc5caaceacf5e392ae46a0ab6dd18f6256eeb2f737 not found: ID does not exist" Aug 04 12:30:10.510058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.509972 2578 scope.go:117] "RemoveContainer" containerID="e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def" Aug 04 12:30:10.510058 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510054 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 04 12:30:10.510379 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510149 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 04 12:30:10.510379 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510203 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 04 12:30:10.510379 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510216 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 04 12:30:10.510379 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510278 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def"} err="failed to get container status \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": rpc error: code = NotFound desc = could not find container \"e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def\": container with ID starting with e409e02b95c6742bf2f4ce69cd2381900d16ce6648221477d1a3e6d6d0e26def not found: ID does not exist" Aug 04 12:30:10.510379 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510299 2578 scope.go:117] "RemoveContainer" containerID="3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b" Aug 04 12:30:10.510657 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510439 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 04 12:30:10.510657 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510452 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r4fwd\"" Aug 04 12:30:10.510657 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510528 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b"} err="failed to get container status \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": rpc error: code = NotFound desc = could not find container \"3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b\": container with ID starting with 3a7f5a29e1f96677e9079d42048671ebb6074ba9665b16838ed5487b83196c9b not found: ID does not exist" Aug 04 12:30:10.510657 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510538 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 04 12:30:10.510657 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510547 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 04 12:30:10.510936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510705 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 04 12:30:10.510936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510776 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 04 12:30:10.510936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.510786 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2bai1au17209d\"" Aug 04 12:30:10.515814 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.515792 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 04 12:30:10.518582 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.518528 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 04 12:30:10.520337 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.520316 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:30:10.617486 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617486 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-web-config\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07229bba-8c2b-493a-b3d2-61d63f6c336e-config-out\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07229bba-8c2b-493a-b3d2-61d63f6c336e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617699 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617782 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wshrm\" (UniqueName: \"kubernetes.io/projected/07229bba-8c2b-493a-b3d2-61d63f6c336e-kube-api-access-wshrm\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-config\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.617936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.618164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.618164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.618164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.617987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.718988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.718898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.718988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.718934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.718988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.718963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.718988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.718987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07229bba-8c2b-493a-b3d2-61d63f6c336e-config-out\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07229bba-8c2b-493a-b3d2-61d63f6c336e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wshrm\" (UniqueName: \"kubernetes.io/projected/07229bba-8c2b-493a-b3d2-61d63f6c336e-kube-api-access-wshrm\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-config\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.719682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-web-config\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.720197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.719801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.720197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.720022 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.720576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.720548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.721204 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.720743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.721204 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.720909 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.721204 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.721003 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.723207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.723116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/07229bba-8c2b-493a-b3d2-61d63f6c336e-config-out\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.723467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/07229bba-8c2b-493a-b3d2-61d63f6c336e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.723648 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.723687 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.723898 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.724142 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-config\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724487 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.724294 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.724823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.724795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/07229bba-8c2b-493a-b3d2-61d63f6c336e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.725078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.725058 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.725591 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.725563 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.725746 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.725730 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.726172 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.726104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/07229bba-8c2b-493a-b3d2-61d63f6c336e-web-config\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.731419 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.731397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wshrm\" (UniqueName: \"kubernetes.io/projected/07229bba-8c2b-493a-b3d2-61d63f6c336e-kube-api-access-wshrm\") pod \"prometheus-k8s-0\" (UID: \"07229bba-8c2b-493a-b3d2-61d63f6c336e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.818089 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.818058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:10.965180 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:10.965149 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 12:30:10.968167 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:30:10.968137 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07229bba_8c2b_493a_b3d2_61d63f6c336e.slice/crio-35e79411bf146bc5787b4cb30865c74e5a5db49230ae1025264c99a2eb2ccd8c WatchSource:0}: Error finding container 35e79411bf146bc5787b4cb30865c74e5a5db49230ae1025264c99a2eb2ccd8c: Status 404 returned error can't find the container with id 35e79411bf146bc5787b4cb30865c74e5a5db49230ae1025264c99a2eb2ccd8c Aug 04 12:30:11.435682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:11.435652 2578 generic.go:356] "Generic (PLEG): container finished" podID="07229bba-8c2b-493a-b3d2-61d63f6c336e" containerID="1b1a6e0ef8aafe5eb96fe49953d7709a52580dc1c2ea743e32773e327cc038d2" exitCode=0 Aug 04 12:30:11.436226 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:11.435740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerDied","Data":"1b1a6e0ef8aafe5eb96fe49953d7709a52580dc1c2ea743e32773e327cc038d2"} Aug 04 12:30:11.436226 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:11.435816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"35e79411bf146bc5787b4cb30865c74e5a5db49230ae1025264c99a2eb2ccd8c"} Aug 04 12:30:11.552822 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:11.552786 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6e4fe5e-5491-4c35-b219-82cc3a217ca8" path="/var/lib/kubelet/pods/b6e4fe5e-5491-4c35-b219-82cc3a217ca8/volumes" Aug 04 12:30:12.444743 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.444706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"ca4c037190abe16f91fcacc356649e1a83534acafa442de29c925694598f9c08"} Aug 04 12:30:12.445185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.444778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"2008e7433f1766d1badd05c828428bb3e2fb3a1c4c33717015c7830da27fc307"} Aug 04 12:30:12.445185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.444806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"bdf1e081df0e17c98589baabce515e9c89ec366173555e51803a041427214000"} Aug 04 12:30:12.445185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.444832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"8c1b57bd295c3a97ccb2ccaea1a047cd31d4e60bccc171c9833ebddaf8f42cdd"} Aug 04 12:30:12.445185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.444858 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"b7a19d13272ca25d40dc9906f549d80e6f8a58b556d8c3726f24df2fb76b775f"} Aug 04 12:30:12.445185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.444883 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"07229bba-8c2b-493a-b3d2-61d63f6c336e","Type":"ContainerStarted","Data":"3c0c337650b389f2a370446ed6639871a68eed7d693c5cbf6dcc65c24281f4be"} Aug 04 12:30:12.486464 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:12.486413 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.466749188 podStartE2EDuration="2.486396645s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.647457ms" isStatefulPod=false podCreationTimestamp="2026-08-04 12:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:30:12.484245099 +0000 UTC m=+143.439840911" watchObservedRunningTime="2026-08-04 12:30:12.486396645 +0000 UTC m=+143.441992439" Aug 04 12:30:14.294807 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.294774 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c77ff8db-x8x85"] Aug 04 12:30:14.299266 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.299245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.336401 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.336373 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c77ff8db-x8x85"] Aug 04 12:30:14.353168 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-console-config\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.353304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-trusted-ca-bundle\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.353304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-service-ca\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.353304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-oauth-config\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.353427 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-oauth-serving-cert\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.353427 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-serving-cert\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.353427 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.353370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdtlf\" (UniqueName: \"kubernetes.io/projected/05a96daf-3d12-4405-a230-c4535cbe59d5-kube-api-access-sdtlf\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.385395 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.385341 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:30:14.454345 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-console-config\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.454520 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-trusted-ca-bundle\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.454520 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-service-ca\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.454520 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-oauth-config\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.454520 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-oauth-serving-cert\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.454520 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-serving-cert\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.454520 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.454480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdtlf\" (UniqueName: \"kubernetes.io/projected/05a96daf-3d12-4405-a230-c4535cbe59d5-kube-api-access-sdtlf\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.455190 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.455163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-console-config\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.455334 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.455200 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-oauth-serving-cert\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.455451 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.455422 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-service-ca\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.455563 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.455493 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-trusted-ca-bundle\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.457039 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.457017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-oauth-config\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.457591 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.457572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-serving-cert\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.464648 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.464630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdtlf\" (UniqueName: \"kubernetes.io/projected/05a96daf-3d12-4405-a230-c4535cbe59d5-kube-api-access-sdtlf\") pod \"console-68c77ff8db-x8x85\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.608862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.608827 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:14.762297 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:14.762269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c77ff8db-x8x85"] Aug 04 12:30:14.764937 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:30:14.764893 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a96daf_3d12_4405_a230_c4535cbe59d5.slice/crio-6bfb4996f06576a122d635b2901fa7d4668cec84cb88a6c0cd9d0993193d5a9b WatchSource:0}: Error finding container 6bfb4996f06576a122d635b2901fa7d4668cec84cb88a6c0cd9d0993193d5a9b: Status 404 returned error can't find the container with id 6bfb4996f06576a122d635b2901fa7d4668cec84cb88a6c0cd9d0993193d5a9b Aug 04 12:30:15.460178 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:15.460140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c77ff8db-x8x85" event={"ID":"05a96daf-3d12-4405-a230-c4535cbe59d5","Type":"ContainerStarted","Data":"9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7"} Aug 04 12:30:15.460628 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:15.460189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c77ff8db-x8x85" event={"ID":"05a96daf-3d12-4405-a230-c4535cbe59d5","Type":"ContainerStarted","Data":"6bfb4996f06576a122d635b2901fa7d4668cec84cb88a6c0cd9d0993193d5a9b"} Aug 04 12:30:15.481543 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:15.481497 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68c77ff8db-x8x85" podStartSLOduration=1.48148048 podStartE2EDuration="1.48148048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:30:15.479771771 +0000 UTC m=+146.435367578" watchObservedRunningTime="2026-08-04 12:30:15.48148048 +0000 UTC m=+146.437076273" Aug 04 12:30:15.819190 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:15.819089 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:30:24.609920 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:24.609873 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:24.610338 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:24.609946 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:24.614700 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:24.614674 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:25.504334 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:25.504303 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:30:25.558869 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:25.558840 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dd7dbf45-qt86n"] Aug 04 12:30:25.732425 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:25.732396 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:30:25.833047 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:25.832967 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:30:50.580866 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.580818 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54dd7dbf45-qt86n" podUID="4c74b51e-2661-40cc-95d0-e2c8ade362e4" containerName="console" containerID="cri-o://e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e" gracePeriod=15 Aug 04 12:30:50.820975 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.820947 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:30:50.885360 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885323 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-oauth-config\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885562 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885413 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-serving-cert\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885562 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885503 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-trusted-ca-bundle\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885562 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-oauth-serving-cert\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885747 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885574 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9hk7\" (UniqueName: \"kubernetes.io/projected/4c74b51e-2661-40cc-95d0-e2c8ade362e4-kube-api-access-z9hk7\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885747 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885611 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-config\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885747 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885656 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-service-ca\") pod \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\" (UID: \"4c74b51e-2661-40cc-95d0-e2c8ade362e4\") " Aug 04 12:30:50.885952 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885925 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:50.886020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885949 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:50.886020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.885963 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-config" (OuterVolumeSpecName: "console-config") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:50.886321 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.886218 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:30:50.887742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.887711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:50.887874 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.887847 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:30:50.887874 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.887865 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c74b51e-2661-40cc-95d0-e2c8ade362e4-kube-api-access-z9hk7" (OuterVolumeSpecName: "kube-api-access-z9hk7") pod "4c74b51e-2661-40cc-95d0-e2c8ade362e4" (UID: "4c74b51e-2661-40cc-95d0-e2c8ade362e4"). InnerVolumeSpecName "kube-api-access-z9hk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:30:50.986597 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986553 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-oauth-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:50.986597 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986590 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:50.986597 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986604 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-trusted-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:50.986851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986617 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-oauth-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:50.986851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986630 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hk7\" (UniqueName: \"kubernetes.io/projected/4c74b51e-2661-40cc-95d0-e2c8ade362e4-kube-api-access-z9hk7\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:50.986851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986644 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-console-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:50.986851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:50.986657 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c74b51e-2661-40cc-95d0-e2c8ade362e4-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:30:51.624950 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.624915 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c74b51e-2661-40cc-95d0-e2c8ade362e4" containerID="e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e" exitCode=0 Aug 04 12:30:51.625377 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.624968 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dd7dbf45-qt86n" Aug 04 12:30:51.625377 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.624979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dd7dbf45-qt86n" event={"ID":"4c74b51e-2661-40cc-95d0-e2c8ade362e4","Type":"ContainerDied","Data":"e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e"} Aug 04 12:30:51.625377 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.625020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dd7dbf45-qt86n" event={"ID":"4c74b51e-2661-40cc-95d0-e2c8ade362e4","Type":"ContainerDied","Data":"22098dabb7e276e76bb805df238d16ed184dc2be59adeedc43ec92fa19d39f34"} Aug 04 12:30:51.625377 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.625038 2578 scope.go:117] "RemoveContainer" containerID="e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e" Aug 04 12:30:51.633164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.633137 2578 scope.go:117] "RemoveContainer" containerID="e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e" Aug 04 12:30:51.633483 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:30:51.633463 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e\": container with ID starting with e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e not found: ID does not exist" containerID="e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e" Aug 04 12:30:51.633555 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.633496 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e"} err="failed to get container status \"e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e\": rpc error: code = NotFound desc = could not find container \"e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e\": container with ID starting with e3197700a3bd787666620df924c1d57eebb1b058e7baec132db3a0352ad9bf7e not found: ID does not exist" Aug 04 12:30:51.645057 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.645032 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dd7dbf45-qt86n"] Aug 04 12:30:51.649783 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:51.649760 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54dd7dbf45-qt86n"] Aug 04 12:30:53.552430 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:53.552394 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c74b51e-2661-40cc-95d0-e2c8ade362e4" path="/var/lib/kubelet/pods/4c74b51e-2661-40cc-95d0-e2c8ade362e4/volumes" Aug 04 12:30:57.339471 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:30:57.339440 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:10.818680 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:10.818635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:31:10.834883 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:10.834856 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:31:11.720558 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:11.720529 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 12:31:19.727090 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:19.727054 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:19.899902 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:19.899873 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:28.529253 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.529219 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:28.828868 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.828784 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74fb998b6c-mjfkj"] Aug 04 12:31:28.829157 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.829144 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c74b51e-2661-40cc-95d0-e2c8ade362e4" containerName="console" Aug 04 12:31:28.829218 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.829159 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c74b51e-2661-40cc-95d0-e2c8ade362e4" containerName="console" Aug 04 12:31:28.829218 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.829213 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c74b51e-2661-40cc-95d0-e2c8ade362e4" containerName="console" Aug 04 12:31:28.833439 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.833415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:28.846016 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.845988 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fb998b6c-mjfkj"] Aug 04 12:31:28.969633 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:28.969583 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:29.022814 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbl6l\" (UniqueName: \"kubernetes.io/projected/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-kube-api-access-hbl6l\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.022814 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-trusted-ca-bundle\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.022985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-oauth-config\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.022985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-config\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.022985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-serving-cert\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.022985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-oauth-serving-cert\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.022985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.022980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-service-ca\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.071501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.071465 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:29.124231 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-oauth-serving-cert\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.124408 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-service-ca\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.124408 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbl6l\" (UniqueName: \"kubernetes.io/projected/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-kube-api-access-hbl6l\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.124408 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-trusted-ca-bundle\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.124671 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-oauth-config\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.124671 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-config\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.124671 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-serving-cert\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.125005 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.124973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-oauth-serving-cert\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.125093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.125015 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-service-ca\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.125208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.125186 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-config\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.125399 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.125238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-trusted-ca-bundle\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.127088 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.127053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-oauth-config\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.127210 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.127175 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-serving-cert\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.134818 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.134795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbl6l\" (UniqueName: \"kubernetes.io/projected/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-kube-api-access-hbl6l\") pod \"console-74fb998b6c-mjfkj\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.142636 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.142619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:29.283926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.283894 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fb998b6c-mjfkj"] Aug 04 12:31:29.286851 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:31:29.286823 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b58c9b_b8bc_4bde_bcb6_54da0d75cdf5.slice/crio-06bf973a3eec4c3bc1a30942d1b9b240e7b37c3182fd316edef6db76154622e6 WatchSource:0}: Error finding container 06bf973a3eec4c3bc1a30942d1b9b240e7b37c3182fd316edef6db76154622e6: Status 404 returned error can't find the container with id 06bf973a3eec4c3bc1a30942d1b9b240e7b37c3182fd316edef6db76154622e6 Aug 04 12:31:29.774738 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.774701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fb998b6c-mjfkj" event={"ID":"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5","Type":"ContainerStarted","Data":"54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4"} Aug 04 12:31:29.775202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.774759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fb998b6c-mjfkj" event={"ID":"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5","Type":"ContainerStarted","Data":"06bf973a3eec4c3bc1a30942d1b9b240e7b37c3182fd316edef6db76154622e6"} Aug 04 12:31:29.798447 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:29.798396 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74fb998b6c-mjfkj" podStartSLOduration=1.798379953 podStartE2EDuration="1.798379953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:31:29.796514378 +0000 UTC m=+220.752110173" watchObservedRunningTime="2026-08-04 12:31:29.798379953 +0000 UTC m=+220.753975746" Aug 04 12:31:31.677714 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:31.677675 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:31.738142 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:31.738095 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:37.184884 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:37.184848 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:38.266597 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:38.266564 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:38.365508 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:38.365476 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:39.143750 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:39.143688 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:39.143945 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:39.143811 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:39.148719 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:39.148697 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:39.819468 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:39.819442 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:31:39.877938 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:39.877909 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c77ff8db-x8x85"] Aug 04 12:31:39.991769 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:39.991718 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:31:40.051027 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:31:40.050995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:32:04.903941 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:04.903879 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68c77ff8db-x8x85" podUID="05a96daf-3d12-4405-a230-c4535cbe59d5" containerName="console" containerID="cri-o://9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7" gracePeriod=15 Aug 04 12:32:05.166457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.166397 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:32:05.250903 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.250865 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-trusted-ca-bundle\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.250913 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-oauth-serving-cert\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.250938 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-console-config\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.250966 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-oauth-config\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251002 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-service-ca\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251020 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdtlf\" (UniqueName: \"kubernetes.io/projected/05a96daf-3d12-4405-a230-c4535cbe59d5-kube-api-access-sdtlf\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-serving-cert\") pod \"05a96daf-3d12-4405-a230-c4535cbe59d5\" (UID: \"05a96daf-3d12-4405-a230-c4535cbe59d5\") " Aug 04 12:32:05.251462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251364 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:32:05.251527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251497 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-console-config" (OuterVolumeSpecName: "console-config") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:32:05.251583 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251523 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:32:05.251583 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.251531 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-service-ca" (OuterVolumeSpecName: "service-ca") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:32:05.253359 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.253325 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:32:05.253480 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.253390 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:32:05.253480 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.253446 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a96daf-3d12-4405-a230-c4535cbe59d5-kube-api-access-sdtlf" (OuterVolumeSpecName: "kube-api-access-sdtlf") pod "05a96daf-3d12-4405-a230-c4535cbe59d5" (UID: "05a96daf-3d12-4405-a230-c4535cbe59d5"). InnerVolumeSpecName "kube-api-access-sdtlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:32:05.352210 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352168 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-console-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.352210 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352199 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-oauth-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.352210 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352211 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.352210 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352219 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdtlf\" (UniqueName: \"kubernetes.io/projected/05a96daf-3d12-4405-a230-c4535cbe59d5-kube-api-access-sdtlf\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.352469 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352228 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05a96daf-3d12-4405-a230-c4535cbe59d5-console-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.352469 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352237 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-trusted-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.352469 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.352246 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05a96daf-3d12-4405-a230-c4535cbe59d5-oauth-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:32:05.918727 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.918696 2578 generic.go:356] "Generic (PLEG): container finished" podID="05a96daf-3d12-4405-a230-c4535cbe59d5" containerID="9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7" exitCode=0 Aug 04 12:32:05.919116 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.918743 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c77ff8db-x8x85" Aug 04 12:32:05.919116 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.918755 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c77ff8db-x8x85" event={"ID":"05a96daf-3d12-4405-a230-c4535cbe59d5","Type":"ContainerDied","Data":"9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7"} Aug 04 12:32:05.919116 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.918796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c77ff8db-x8x85" event={"ID":"05a96daf-3d12-4405-a230-c4535cbe59d5","Type":"ContainerDied","Data":"6bfb4996f06576a122d635b2901fa7d4668cec84cb88a6c0cd9d0993193d5a9b"} Aug 04 12:32:05.919116 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.918815 2578 scope.go:117] "RemoveContainer" containerID="9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7" Aug 04 12:32:05.927353 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.927335 2578 scope.go:117] "RemoveContainer" containerID="9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7" Aug 04 12:32:05.927614 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:32:05.927595 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7\": container with ID starting with 9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7 not found: ID does not exist" containerID="9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7" Aug 04 12:32:05.927667 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.927623 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7"} err="failed to get container status \"9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7\": rpc error: code = NotFound desc = could not find container \"9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7\": container with ID starting with 9782ff1bc37df4680e1858ef8267e8666830e43dc647006dd507d77568f35af7 not found: ID does not exist" Aug 04 12:32:05.938670 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.938639 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c77ff8db-x8x85"] Aug 04 12:32:05.943257 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:05.943237 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c77ff8db-x8x85"] Aug 04 12:32:07.552018 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:07.551983 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a96daf-3d12-4405-a230-c4535cbe59d5" path="/var/lib/kubelet/pods/05a96daf-3d12-4405-a230-c4535cbe59d5/volumes" Aug 04 12:32:42.835495 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:42.835461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:32:49.390307 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:49.390277 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:32:49.393969 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:49.393947 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:32:49.397878 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:49.397858 2578 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 12:32:51.775531 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:51.775492 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:32:51.870472 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:32:51.870430 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:18.507951 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:18.507913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:19.272346 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:19.272309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:22.336853 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:22.336816 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:22.436008 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:22.435970 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:24.697922 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:24.697840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:24.813387 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:24.813351 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:25.191108 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:25.191071 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:25.287690 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:25.287649 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:27.572401 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:27.572338 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:27.788875 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:27.788745 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:28.720075 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.720030 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:28.752885 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.752851 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x444m"] Aug 04 12:33:28.753202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.753189 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a96daf-3d12-4405-a230-c4535cbe59d5" containerName="console" Aug 04 12:33:28.753202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.753203 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a96daf-3d12-4405-a230-c4535cbe59d5" containerName="console" Aug 04 12:33:28.753295 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.753256 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a96daf-3d12-4405-a230-c4535cbe59d5" containerName="console" Aug 04 12:33:28.756077 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.756062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:28.758717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.758692 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-87p2g\"" Aug 04 12:33:28.758717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.758709 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 12:33:28.758906 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.758702 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 12:33:28.765979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.765957 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x444m"] Aug 04 12:33:28.781571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.781546 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:28.854643 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.854605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qgr\" (UniqueName: \"kubernetes.io/projected/9d4fe818-f821-478d-8ec3-0bde4de50961-kube-api-access-98qgr\") pod \"cert-manager-cainjector-559cdd8c87-x444m\" (UID: \"9d4fe818-f821-478d-8ec3-0bde4de50961\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:28.854832 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.854671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4fe818-f821-478d-8ec3-0bde4de50961-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x444m\" (UID: \"9d4fe818-f821-478d-8ec3-0bde4de50961\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:28.955399 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.955361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qgr\" (UniqueName: \"kubernetes.io/projected/9d4fe818-f821-478d-8ec3-0bde4de50961-kube-api-access-98qgr\") pod \"cert-manager-cainjector-559cdd8c87-x444m\" (UID: \"9d4fe818-f821-478d-8ec3-0bde4de50961\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:28.955572 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.955420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4fe818-f821-478d-8ec3-0bde4de50961-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x444m\" (UID: \"9d4fe818-f821-478d-8ec3-0bde4de50961\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:28.966689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.966657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qgr\" (UniqueName: \"kubernetes.io/projected/9d4fe818-f821-478d-8ec3-0bde4de50961-kube-api-access-98qgr\") pod \"cert-manager-cainjector-559cdd8c87-x444m\" (UID: \"9d4fe818-f821-478d-8ec3-0bde4de50961\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:28.968264 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:28.968244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d4fe818-f821-478d-8ec3-0bde4de50961-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x444m\" (UID: \"9d4fe818-f821-478d-8ec3-0bde4de50961\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:29.077395 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:29.077298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" Aug 04 12:33:29.214320 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:29.214287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x444m"] Aug 04 12:33:29.218178 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:33:29.218137 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d4fe818_f821_478d_8ec3_0bde4de50961.slice/crio-a4a0447bd0ba647dfe0825752230c036d2418a00dceac797cb6f8aaf92ca80f7 WatchSource:0}: Error finding container a4a0447bd0ba647dfe0825752230c036d2418a00dceac797cb6f8aaf92ca80f7: Status 404 returned error can't find the container with id a4a0447bd0ba647dfe0825752230c036d2418a00dceac797cb6f8aaf92ca80f7 Aug 04 12:33:29.220643 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:29.220626 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 12:33:29.255782 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:29.255750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" event={"ID":"9d4fe818-f821-478d-8ec3-0bde4de50961","Type":"ContainerStarted","Data":"a4a0447bd0ba647dfe0825752230c036d2418a00dceac797cb6f8aaf92ca80f7"} Aug 04 12:33:32.273065 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:32.272968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" event={"ID":"9d4fe818-f821-478d-8ec3-0bde4de50961","Type":"ContainerStarted","Data":"19c7ec8b70966474b13580a4d314d1c69e97e7812a9f9661e5819f1c66f54e3e"} Aug 04 12:33:32.296271 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:32.296222 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x444m" podStartSLOduration=1.519302042 podStartE2EDuration="4.296203335s" totalImagesPullingTime="2.776901293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:33:32.295071551 +0000 UTC m=+343.250667355" watchObservedRunningTime="2026-08-04 12:33:32.296203335 +0000 UTC m=+343.251799128" Aug 04 12:33:32.780374 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:32.780340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:32.881272 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:32.881239 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:34.788075 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:34.788045 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:36.289958 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.289928 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l2qm7"] Aug 04 12:33:36.293262 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.293246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.296311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.296294 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-swxv2\"" Aug 04 12:33:36.302105 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.302084 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l2qm7"] Aug 04 12:33:36.424017 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.423982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddkqj\" (UniqueName: \"kubernetes.io/projected/585d0f8c-f746-409f-bb9e-abbedff51f14-kube-api-access-ddkqj\") pod \"cert-manager-55b4d954d5-l2qm7\" (UID: \"585d0f8c-f746-409f-bb9e-abbedff51f14\") " pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.424212 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.424036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/585d0f8c-f746-409f-bb9e-abbedff51f14-bound-sa-token\") pod \"cert-manager-55b4d954d5-l2qm7\" (UID: \"585d0f8c-f746-409f-bb9e-abbedff51f14\") " pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.525350 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.525315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/585d0f8c-f746-409f-bb9e-abbedff51f14-bound-sa-token\") pod \"cert-manager-55b4d954d5-l2qm7\" (UID: \"585d0f8c-f746-409f-bb9e-abbedff51f14\") " pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.525512 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.525392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddkqj\" (UniqueName: \"kubernetes.io/projected/585d0f8c-f746-409f-bb9e-abbedff51f14-kube-api-access-ddkqj\") pod \"cert-manager-55b4d954d5-l2qm7\" (UID: \"585d0f8c-f746-409f-bb9e-abbedff51f14\") " pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.535095 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.535057 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/585d0f8c-f746-409f-bb9e-abbedff51f14-bound-sa-token\") pod \"cert-manager-55b4d954d5-l2qm7\" (UID: \"585d0f8c-f746-409f-bb9e-abbedff51f14\") " pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.535225 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.535167 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddkqj\" (UniqueName: \"kubernetes.io/projected/585d0f8c-f746-409f-bb9e-abbedff51f14-kube-api-access-ddkqj\") pod \"cert-manager-55b4d954d5-l2qm7\" (UID: \"585d0f8c-f746-409f-bb9e-abbedff51f14\") " pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.601977 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.601950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l2qm7" Aug 04 12:33:36.714692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.714660 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d9497cb75-fzgk8"] Aug 04 12:33:36.719068 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.719046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.729928 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.729900 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9497cb75-fzgk8"] Aug 04 12:33:36.732631 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.732608 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l2qm7"] Aug 04 12:33:36.736286 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:33:36.736262 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod585d0f8c_f746_409f_bb9e_abbedff51f14.slice/crio-046fbbdf1c878f775cb6c6784f9440fde74c5979b03b1da8d0ce5e90bcadc133 WatchSource:0}: Error finding container 046fbbdf1c878f775cb6c6784f9440fde74c5979b03b1da8d0ce5e90bcadc133: Status 404 returned error can't find the container with id 046fbbdf1c878f775cb6c6784f9440fde74c5979b03b1da8d0ce5e90bcadc133 Aug 04 12:33:36.772367 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.772345 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:36.828258 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-oauth-serving-cert\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.828437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-trusted-ca-bundle\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.828437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-service-ca\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.828437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cjtk\" (UniqueName: \"kubernetes.io/projected/d3529031-70d2-45fa-964e-1f0664f7edd6-kube-api-access-6cjtk\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.828641 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3529031-70d2-45fa-964e-1f0664f7edd6-console-oauth-config\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.828641 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-console-config\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.828641 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.828512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3529031-70d2-45fa-964e-1f0664f7edd6-console-serving-cert\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.882318 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.882239 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:36.928932 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.928906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-oauth-serving-cert\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929099 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.928940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-trusted-ca-bundle\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929099 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.928967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-service-ca\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929099 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cjtk\" (UniqueName: \"kubernetes.io/projected/d3529031-70d2-45fa-964e-1f0664f7edd6-kube-api-access-6cjtk\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3529031-70d2-45fa-964e-1f0664f7edd6-console-oauth-config\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-console-config\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929311 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3529031-70d2-45fa-964e-1f0664f7edd6-console-serving-cert\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929714 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929686 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-service-ca\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929880 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929858 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-trusted-ca-bundle\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.929967 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.929927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-oauth-serving-cert\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.930221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.930189 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3529031-70d2-45fa-964e-1f0664f7edd6-console-config\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.931726 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.931710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3529031-70d2-45fa-964e-1f0664f7edd6-console-oauth-config\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.931831 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.931813 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3529031-70d2-45fa-964e-1f0664f7edd6-console-serving-cert\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.938437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.938417 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cjtk\" (UniqueName: \"kubernetes.io/projected/d3529031-70d2-45fa-964e-1f0664f7edd6-kube-api-access-6cjtk\") pod \"console-d9497cb75-fzgk8\" (UID: \"d3529031-70d2-45fa-964e-1f0664f7edd6\") " pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:36.966942 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:36.966920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:37.029236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.029200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:37.161582 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.161544 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9497cb75-fzgk8"] Aug 04 12:33:37.164161 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:33:37.164115 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3529031_70d2_45fa_964e_1f0664f7edd6.slice/crio-b5a485a9cf9f6752d7158997dcfd8da52b74e6c5a4338622011db86503a747f8 WatchSource:0}: Error finding container b5a485a9cf9f6752d7158997dcfd8da52b74e6c5a4338622011db86503a747f8: Status 404 returned error can't find the container with id b5a485a9cf9f6752d7158997dcfd8da52b74e6c5a4338622011db86503a747f8 Aug 04 12:33:37.294907 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.294869 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l2qm7" event={"ID":"585d0f8c-f746-409f-bb9e-abbedff51f14","Type":"ContainerStarted","Data":"04fc696b6b1ed0b07234b78c3f1458306c39bd4022f8af8dc3487568c0bffed0"} Aug 04 12:33:37.295462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.294920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l2qm7" event={"ID":"585d0f8c-f746-409f-bb9e-abbedff51f14","Type":"ContainerStarted","Data":"046fbbdf1c878f775cb6c6784f9440fde74c5979b03b1da8d0ce5e90bcadc133"} Aug 04 12:33:37.296573 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.296549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9497cb75-fzgk8" event={"ID":"d3529031-70d2-45fa-964e-1f0664f7edd6","Type":"ContainerStarted","Data":"c245c373512caedef188e4e7fd661297d5b3f0618f1d3ea95a2071109f8efe17"} Aug 04 12:33:37.296713 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.296587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9497cb75-fzgk8" event={"ID":"d3529031-70d2-45fa-964e-1f0664f7edd6","Type":"ContainerStarted","Data":"b5a485a9cf9f6752d7158997dcfd8da52b74e6c5a4338622011db86503a747f8"} Aug 04 12:33:37.313993 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.313946 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-l2qm7" podStartSLOduration=1.31393229 podStartE2EDuration="1.31393229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:33:37.313148458 +0000 UTC m=+348.268744246" watchObservedRunningTime="2026-08-04 12:33:37.31393229 +0000 UTC m=+348.269528083" Aug 04 12:33:37.338395 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.338336 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d9497cb75-fzgk8" podStartSLOduration=1.33831621 podStartE2EDuration="1.33831621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:33:37.336912783 +0000 UTC m=+348.292508592" watchObservedRunningTime="2026-08-04 12:33:37.33831621 +0000 UTC m=+348.293912006" Aug 04 12:33:37.743623 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:37.743571 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:38.709184 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:38.709150 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:38.805741 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:38.805706 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:45.114036 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:45.113968 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:45.215290 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:45.215250 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:46.884672 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:46.884639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:47.029607 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:47.029565 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:47.029815 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:47.029632 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:47.034393 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:47.034369 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:47.339489 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:47.339469 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d9497cb75-fzgk8" Aug 04 12:33:47.393111 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:47.393080 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fb998b6c-mjfkj"] Aug 04 12:33:48.076546 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:48.076508 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:48.134428 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:48.134397 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:51.509221 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:51.509185 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:51.571763 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:51.571727 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:53.458799 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:53.458755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:53.524825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:53.524790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:57.108283 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.108258 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:57.150426 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.150394 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8"] Aug 04 12:33:57.157286 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.156710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.160420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.160395 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 12:33:57.160571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.160550 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 12:33:57.160633 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.160620 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 12:33:57.160761 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.160640 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 12:33:57.166331 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.166315 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2pxdk\"" Aug 04 12:33:57.170243 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.170223 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 12:33:57.174414 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.174391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8"] Aug 04 12:33:57.200238 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.200205 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:33:57.207548 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.207524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvkc\" (UniqueName: \"kubernetes.io/projected/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-kube-api-access-ldvkc\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.207693 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.207558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-metrics-cert\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.207693 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.207595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-manager-config\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.207693 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.207629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-cert\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.308072 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.308035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-cert\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.308293 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.308107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvkc\" (UniqueName: \"kubernetes.io/projected/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-kube-api-access-ldvkc\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.308293 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.308159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-metrics-cert\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.308293 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.308209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-manager-config\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.309111 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.309087 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-manager-config\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.310854 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.310831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-cert\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.310992 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.310923 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-metrics-cert\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.318398 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.318370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvkc\" (UniqueName: \"kubernetes.io/projected/ed7cdc18-0e8e-47b0-86ab-1348fb415be5-kube-api-access-ldvkc\") pod \"lws-controller-manager-c4cd7f74c-8dgn8\" (UID: \"ed7cdc18-0e8e-47b0-86ab-1348fb415be5\") " pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.466922 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.466831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:33:57.618443 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:57.618413 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8"] Aug 04 12:33:57.621761 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:33:57.621735 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7cdc18_0e8e_47b0_86ab_1348fb415be5.slice/crio-530a281541fdc58b37bd4170426bfb4a1901a29b48fd42cd38a57ca9127dbff3 WatchSource:0}: Error finding container 530a281541fdc58b37bd4170426bfb4a1901a29b48fd42cd38a57ca9127dbff3: Status 404 returned error can't find the container with id 530a281541fdc58b37bd4170426bfb4a1901a29b48fd42cd38a57ca9127dbff3 Aug 04 12:33:58.381651 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:33:58.381612 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" event={"ID":"ed7cdc18-0e8e-47b0-86ab-1348fb415be5","Type":"ContainerStarted","Data":"530a281541fdc58b37bd4170426bfb4a1901a29b48fd42cd38a57ca9127dbff3"} Aug 04 12:34:01.396969 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:01.396939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" event={"ID":"ed7cdc18-0e8e-47b0-86ab-1348fb415be5","Type":"ContainerStarted","Data":"56fa01aebb63877b6c5a6544825c58e86d27dfd5da5e9241383ee31f38d76f21"} Aug 04 12:34:01.397367 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:01.397067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:34:01.416689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:01.416650 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" podStartSLOduration=1.6378371 podStartE2EDuration="4.416636345s" totalImagesPullingTime="2.778799245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:34:01.415714154 +0000 UTC m=+372.371309949" watchObservedRunningTime="2026-08-04 12:34:01.416636345 +0000 UTC m=+372.372232138" Aug 04 12:34:01.504440 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:01.504405 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:01.606267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:01.606232 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:02.375560 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.375526 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:02.487627 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.487593 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:02.543141 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.543095 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc"] Aug 04 12:34:02.546903 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.546878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.550314 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.550292 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gzmtm\"" Aug 04 12:34:02.550452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.550295 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 12:34:02.550452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.550386 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 12:34:02.550452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.550431 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 12:34:02.550680 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.550665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 12:34:02.559028 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.558993 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc"] Aug 04 12:34:02.652957 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.652872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3654256c-6789-438c-97f0-08d6c2ef0be2-manifests\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.652957 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.652917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3654256c-6789-438c-97f0-08d6c2ef0be2-tmp\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.653256 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.653097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3654256c-6789-438c-97f0-08d6c2ef0be2-webhook-cert\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.653256 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.653159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsb5d\" (UniqueName: \"kubernetes.io/projected/3654256c-6789-438c-97f0-08d6c2ef0be2-kube-api-access-hsb5d\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.653361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.653259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3654256c-6789-438c-97f0-08d6c2ef0be2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.660708 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.660687 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:02.754207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.754170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3654256c-6789-438c-97f0-08d6c2ef0be2-manifests\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.754207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.754212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3654256c-6789-438c-97f0-08d6c2ef0be2-tmp\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.754435 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.754264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3654256c-6789-438c-97f0-08d6c2ef0be2-webhook-cert\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.754435 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.754286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsb5d\" (UniqueName: \"kubernetes.io/projected/3654256c-6789-438c-97f0-08d6c2ef0be2-kube-api-access-hsb5d\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.754435 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.754309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3654256c-6789-438c-97f0-08d6c2ef0be2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.754692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.754668 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3654256c-6789-438c-97f0-08d6c2ef0be2-manifests\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.756644 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.756613 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3654256c-6789-438c-97f0-08d6c2ef0be2-tmp\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.757000 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.756979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3654256c-6789-438c-97f0-08d6c2ef0be2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.757042 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.756979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3654256c-6789-438c-97f0-08d6c2ef0be2-webhook-cert\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.760808 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.760775 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:02.763644 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.763623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsb5d\" (UniqueName: \"kubernetes.io/projected/3654256c-6789-438c-97f0-08d6c2ef0be2-kube-api-access-hsb5d\") pod \"opendatahub-operator-controller-manager-547f69df5b-9lrrc\" (UID: \"3654256c-6789-438c-97f0-08d6c2ef0be2\") " pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.858758 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.858722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:02.863196 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.863171 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:02.961138 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:02.961086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:03.001865 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:03.001814 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc"] Aug 04 12:34:03.004203 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:34:03.004181 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3654256c_6789_438c_97f0_08d6c2ef0be2.slice/crio-89a6fc9653702f9212fa2963f24bd9fea42f85b9ed6c452db6fbb13a45cb1780 WatchSource:0}: Error finding container 89a6fc9653702f9212fa2963f24bd9fea42f85b9ed6c452db6fbb13a45cb1780: Status 404 returned error can't find the container with id 89a6fc9653702f9212fa2963f24bd9fea42f85b9ed6c452db6fbb13a45cb1780 Aug 04 12:34:03.043822 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:03.043793 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:03.129331 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:03.129296 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:03.382434 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:03.382401 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:03.406519 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:03.406481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" event={"ID":"3654256c-6789-438c-97f0-08d6c2ef0be2","Type":"ContainerStarted","Data":"89a6fc9653702f9212fa2963f24bd9fea42f85b9ed6c452db6fbb13a45cb1780"} Aug 04 12:34:06.423107 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:06.423073 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" event={"ID":"3654256c-6789-438c-97f0-08d6c2ef0be2","Type":"ContainerStarted","Data":"55e019d969e52e074189955a612e8c3ad49d67f0b76259951c491c81be9d5f9a"} Aug 04 12:34:07.430669 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:07.430620 2578 generic.go:356] "Generic (PLEG): container finished" podID="3654256c-6789-438c-97f0-08d6c2ef0be2" containerID="55e019d969e52e074189955a612e8c3ad49d67f0b76259951c491c81be9d5f9a" exitCode=0 Aug 04 12:34:07.431190 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:07.430786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" event={"ID":"3654256c-6789-438c-97f0-08d6c2ef0be2","Type":"ContainerDied","Data":"55e019d969e52e074189955a612e8c3ad49d67f0b76259951c491c81be9d5f9a"} Aug 04 12:34:08.437248 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:08.437213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" event={"ID":"3654256c-6789-438c-97f0-08d6c2ef0be2","Type":"ContainerStarted","Data":"2c5e030d474abd31fffa47589603213eb8a94c8eebedab706586df19e423502e"} Aug 04 12:34:08.437639 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:08.437314 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:08.462443 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:08.462392 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" podStartSLOduration=1.8290657750000001 podStartE2EDuration="6.462376166s" totalImagesPullingTime="3.566662316s" totalInitContainerRuntime="1.066648075s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:34:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:34:08.460911071 +0000 UTC m=+379.416506877" watchObservedRunningTime="2026-08-04 12:34:08.462376166 +0000 UTC m=+379.417971959" Aug 04 12:34:12.403275 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.403241 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c4cd7f74c-8dgn8" Aug 04 12:34:12.413922 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.413886 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74fb998b6c-mjfkj" podUID="71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" containerName="console" containerID="cri-o://54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4" gracePeriod=15 Aug 04 12:34:12.666380 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.666313 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:34:12.749150 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749090 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-trusted-ca-bundle\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749327 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749171 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-config\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749327 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749295 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-serving-cert\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749473 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749378 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-oauth-serving-cert\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749473 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-oauth-config\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbl6l\" (UniqueName: \"kubernetes.io/projected/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-kube-api-access-hbl6l\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749510 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-config" (OuterVolumeSpecName: "console-config") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:34:12.749576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749531 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-service-ca\") pod \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\" (UID: \"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5\") " Aug 04 12:34:12.749787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749532 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:34:12.749787 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749742 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:34:12.749970 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749945 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-service-ca" (OuterVolumeSpecName: "service-ca") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:34:12.750040 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.749967 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-trusted-ca-bundle\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:12.750082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.750013 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:12.750242 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.750105 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-oauth-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:12.751648 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.751619 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-kube-api-access-hbl6l" (OuterVolumeSpecName: "kube-api-access-hbl6l") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "kube-api-access-hbl6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:34:12.751770 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.751719 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:34:12.751770 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.751743 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" (UID: "71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:34:12.850684 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.850647 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-serving-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:12.850684 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.850677 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-console-oauth-config\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:12.850684 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.850689 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbl6l\" (UniqueName: \"kubernetes.io/projected/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-kube-api-access-hbl6l\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:12.850909 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:12.850701 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:34:13.469272 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.469237 2578 generic.go:356] "Generic (PLEG): container finished" podID="71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" containerID="54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4" exitCode=0 Aug 04 12:34:13.469694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.469284 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fb998b6c-mjfkj" Aug 04 12:34:13.469694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.469324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fb998b6c-mjfkj" event={"ID":"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5","Type":"ContainerDied","Data":"54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4"} Aug 04 12:34:13.469694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.469401 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fb998b6c-mjfkj" event={"ID":"71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5","Type":"ContainerDied","Data":"06bf973a3eec4c3bc1a30942d1b9b240e7b37c3182fd316edef6db76154622e6"} Aug 04 12:34:13.469694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.469421 2578 scope.go:117] "RemoveContainer" containerID="54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4" Aug 04 12:34:13.478591 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.478571 2578 scope.go:117] "RemoveContainer" containerID="54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4" Aug 04 12:34:13.478893 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:34:13.478874 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4\": container with ID starting with 54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4 not found: ID does not exist" containerID="54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4" Aug 04 12:34:13.478954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.478903 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4"} err="failed to get container status \"54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4\": rpc error: code = NotFound desc = could not find container \"54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4\": container with ID starting with 54cbd902ecc11c7b33aedf56e9ab41232d7899b484beeb24974f68b7f29b85e4 not found: ID does not exist" Aug 04 12:34:13.492694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.492669 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fb998b6c-mjfkj"] Aug 04 12:34:13.497203 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.497177 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74fb998b6c-mjfkj"] Aug 04 12:34:13.552780 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:13.552735 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" path="/var/lib/kubelet/pods/71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5/volumes" Aug 04 12:34:19.443737 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:19.443706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-547f69df5b-9lrrc" Aug 04 12:34:22.059598 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.059555 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:22.163175 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.163139 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:22.776240 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.776208 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz"] Aug 04 12:34:22.776567 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.776553 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" containerName="console" Aug 04 12:34:22.776615 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.776568 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" containerName="console" Aug 04 12:34:22.776650 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.776639 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b58c9b-b8bc-4bde-bcb6-54da0d75cdf5" containerName="console" Aug 04 12:34:22.788954 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.788931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:22.792187 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.792143 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 12:34:22.792357 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.792151 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xdx9t\"" Aug 04 12:34:22.792855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.792831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz"] Aug 04 12:34:22.793418 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.793398 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 12:34:22.793529 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.793441 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 12:34:22.793529 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.793480 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 12:34:22.943798 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.943771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-tmp\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:22.943969 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.943822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-tls-certs\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:22.943969 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:22.943877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslpn\" (UniqueName: \"kubernetes.io/projected/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-kube-api-access-kslpn\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.010892 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.010854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:23.044718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.044630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-tmp\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.044718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.044698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-tls-certs\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.044925 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.044744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslpn\" (UniqueName: \"kubernetes.io/projected/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-kube-api-access-kslpn\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.047197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.047166 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-tmp\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.047329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.047264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-tls-certs\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.056166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.056114 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslpn\" (UniqueName: \"kubernetes.io/projected/a3bce0e2-a1cc-4759-a048-cf707c9ffe66-kube-api-access-kslpn\") pod \"kube-auth-proxy-5688595d9d-hvkqz\" (UID: \"a3bce0e2-a1cc-4759-a048-cf707c9ffe66\") " pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.100071 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.100035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" Aug 04 12:34:23.108926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.108891 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:23.257082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.257033 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz"] Aug 04 12:34:23.261177 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:34:23.261115 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bce0e2_a1cc_4759_a048_cf707c9ffe66.slice/crio-f13913901fad78e558123cb1c5ec3587935fae28e1b17311ccf177a2b86e4276 WatchSource:0}: Error finding container f13913901fad78e558123cb1c5ec3587935fae28e1b17311ccf177a2b86e4276: Status 404 returned error can't find the container with id f13913901fad78e558123cb1c5ec3587935fae28e1b17311ccf177a2b86e4276 Aug 04 12:34:23.515580 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.515543 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" event={"ID":"a3bce0e2-a1cc-4759-a048-cf707c9ffe66","Type":"ContainerStarted","Data":"f13913901fad78e558123cb1c5ec3587935fae28e1b17311ccf177a2b86e4276"} Aug 04 12:34:23.694438 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.694406 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:23.765487 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:23.765448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:24.028322 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:24.028283 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:24.109113 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:24.109066 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:24.186947 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:24.186908 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:25.815145 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:25.815088 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:26.532964 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:26.532932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" event={"ID":"a3bce0e2-a1cc-4759-a048-cf707c9ffe66","Type":"ContainerStarted","Data":"7abc8fd55e2ce9fab3da2f3e19aae065eb0b4cdf93dc0604b8c8da48f84159c7"} Aug 04 12:34:26.558424 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:26.558371 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5688595d9d-hvkqz" podStartSLOduration=1.821797002 podStartE2EDuration="4.558356568s" totalImagesPullingTime="2.736559566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:34:26.556352203 +0000 UTC m=+397.511947998" watchObservedRunningTime="2026-08-04 12:34:26.558356568 +0000 UTC m=+397.513952362" Aug 04 12:34:30.670901 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:30.670870 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:30.759370 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:30.759336 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:31.964108 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:31.964072 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:32.413715 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:32.413683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:32.523446 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:32.523410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:33.569881 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:33.569843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:33.673626 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:33.673592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:33.761507 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:33.761477 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:33.918496 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:33.918469 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:34.012885 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:34.012851 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:34.116330 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:34.116297 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:34.264990 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:34.264903 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:34.349371 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:34.349340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:34:52.908514 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:34:52.908472 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:23.816635 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:23.816557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:23.917467 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:23.917430 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:36.208861 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:36.208798 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:38.988031 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:38.987996 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:40.575785 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:40.575694 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:40.801008 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:40.800979 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:40.971043 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:40.971010 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:41.076238 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:41.076191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:42.203051 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:42.203020 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:43.588304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:43.588272 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:43.669979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:43.669948 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:44.222986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:44.222915 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:44.468492 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:44.468456 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:44.567518 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:44.567442 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:44.679737 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:44.679704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:49.817041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:49.817007 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:49.920615 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:49.920572 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:51.626422 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:51.626381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:51.729910 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:51.729874 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:54.673491 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.673461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:54.681388 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.681353 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks"] Aug 04 12:35:54.683827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.683809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.686687 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.686659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 04 12:35:54.686775 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.686709 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 12:35:54.688323 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.688301 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 04 12:35:54.688323 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.688301 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rvx5k\"" Aug 04 12:35:54.688466 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.688344 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 12:35:54.693926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.693904 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks"] Aug 04 12:35:54.864244 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.864206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c011127a-ba86-472f-bf21-4585f9e9e1d4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.864488 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.864252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7wzc\" (UniqueName: \"kubernetes.io/projected/c011127a-ba86-472f-bf21-4585f9e9e1d4-kube-api-access-g7wzc\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.864488 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.864329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c011127a-ba86-472f-bf21-4585f9e9e1d4-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.934629 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.934554 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:54.965433 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.965404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c011127a-ba86-472f-bf21-4585f9e9e1d4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.965599 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.965454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wzc\" (UniqueName: \"kubernetes.io/projected/c011127a-ba86-472f-bf21-4585f9e9e1d4-kube-api-access-g7wzc\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.965599 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.965506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c011127a-ba86-472f-bf21-4585f9e9e1d4-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.966254 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.966235 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c011127a-ba86-472f-bf21-4585f9e9e1d4-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.968045 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.968018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c011127a-ba86-472f-bf21-4585f9e9e1d4-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.975735 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.975707 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wzc\" (UniqueName: \"kubernetes.io/projected/c011127a-ba86-472f-bf21-4585f9e9e1d4-kube-api-access-g7wzc\") pod \"kuadrant-console-plugin-598f86468b-vcpks\" (UID: \"c011127a-ba86-472f-bf21-4585f9e9e1d4\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:54.994674 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:54.994647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" Aug 04 12:35:55.129407 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:55.129380 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks"] Aug 04 12:35:55.132631 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:35:55.132601 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc011127a_ba86_472f_bf21_4585f9e9e1d4.slice/crio-0f50c2e5aa21b83657f7e689bab430faab5c86636c2d9c97cb31933c86a55f20 WatchSource:0}: Error finding container 0f50c2e5aa21b83657f7e689bab430faab5c86636c2d9c97cb31933c86a55f20: Status 404 returned error can't find the container with id 0f50c2e5aa21b83657f7e689bab430faab5c86636c2d9c97cb31933c86a55f20 Aug 04 12:35:55.937485 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:55.937450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" event={"ID":"c011127a-ba86-472f-bf21-4585f9e9e1d4","Type":"ContainerStarted","Data":"0f50c2e5aa21b83657f7e689bab430faab5c86636c2d9c97cb31933c86a55f20"} Aug 04 12:35:58.907961 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:58.907890 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:59.008343 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:59.008306 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:35:59.960139 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:59.960089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" event={"ID":"c011127a-ba86-472f-bf21-4585f9e9e1d4","Type":"ContainerStarted","Data":"197bc9245f4cb251a9c2e21f8708dad5e12b67f7afcceabb7c6ad96313d357fd"} Aug 04 12:35:59.978268 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:35:59.978218 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vcpks" podStartSLOduration=1.350392003 podStartE2EDuration="5.978200381s" totalImagesPullingTime="4.627808378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:35:59.977253718 +0000 UTC m=+490.932849513" watchObservedRunningTime="2026-08-04 12:35:59.978200381 +0000 UTC m=+490.933796157" Aug 04 12:36:02.354452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:02.354415 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:02.451598 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:02.451562 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:02.551046 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:02.551015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:09.009656 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:09.009617 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:09.118273 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:09.118240 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:11.991881 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:11.991843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:12.092574 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:12.092543 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:17.827049 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:17.827013 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:17.900287 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:17.900250 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:18.511293 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:18.511257 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:18.592432 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:18.592405 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:21.896678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:21.896647 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:21.995539 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:21.995503 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:22.093223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:22.093194 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:28.636972 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:28.636936 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:34.675715 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:34.675686 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:34.778709 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:34.778676 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:51.738760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.738727 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:51.739567 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.739547 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:36:51.742956 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.742941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:51.745869 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.745851 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 12:36:51.751813 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.751791 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:36:51.836355 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.836320 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:51.849557 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.849531 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:36:51.858206 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.858177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d755fba-ee33-4897-a260-90a53b5ed7c4-config-file\") pod \"limitador-limitador-859c4d96d6-7j8rp\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:51.858315 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.858275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9zq\" (UniqueName: \"kubernetes.io/projected/2d755fba-ee33-4897-a260-90a53b5ed7c4-kube-api-access-nr9zq\") pod \"limitador-limitador-859c4d96d6-7j8rp\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:51.938249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.938220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:51.959029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.958992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d755fba-ee33-4897-a260-90a53b5ed7c4-config-file\") pod \"limitador-limitador-859c4d96d6-7j8rp\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:51.959302 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.959054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9zq\" (UniqueName: \"kubernetes.io/projected/2d755fba-ee33-4897-a260-90a53b5ed7c4-kube-api-access-nr9zq\") pod \"limitador-limitador-859c4d96d6-7j8rp\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:51.959956 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.959932 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d755fba-ee33-4897-a260-90a53b5ed7c4-config-file\") pod \"limitador-limitador-859c4d96d6-7j8rp\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:51.970093 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:51.970068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9zq\" (UniqueName: \"kubernetes.io/projected/2d755fba-ee33-4897-a260-90a53b5ed7c4-kube-api-access-nr9zq\") pod \"limitador-limitador-859c4d96d6-7j8rp\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:52.054189 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.054078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:52.195716 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.195616 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:36:52.198304 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:36:52.198276 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d755fba_ee33_4897_a260_90a53b5ed7c4.slice/crio-e40a3aff9f3ac0cb64adc59c3bd81b23ef89c50e78738d29eba7a7aba792d044 WatchSource:0}: Error finding container e40a3aff9f3ac0cb64adc59c3bd81b23ef89c50e78738d29eba7a7aba792d044: Status 404 returned error can't find the container with id e40a3aff9f3ac0cb64adc59c3bd81b23ef89c50e78738d29eba7a7aba792d044 Aug 04 12:36:52.701543 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.701512 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:52.802749 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.802710 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:36:52.824013 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.823982 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ssjvm"] Aug 04 12:36:52.833081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.833047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:36:52.835300 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.835268 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ssjvm"] Aug 04 12:36:52.836576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.836547 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kb44x\"" Aug 04 12:36:52.866830 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.866784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfsq\" (UniqueName: \"kubernetes.io/projected/c81682b7-5355-4df1-96dd-d558c475c7a7-kube-api-access-bcfsq\") pod \"authorino-c548b5d7d-ssjvm\" (UID: \"c81682b7-5355-4df1-96dd-d558c475c7a7\") " pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:36:52.967407 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.967311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfsq\" (UniqueName: \"kubernetes.io/projected/c81682b7-5355-4df1-96dd-d558c475c7a7-kube-api-access-bcfsq\") pod \"authorino-c548b5d7d-ssjvm\" (UID: \"c81682b7-5355-4df1-96dd-d558c475c7a7\") " pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:36:52.978239 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:52.978196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfsq\" (UniqueName: \"kubernetes.io/projected/c81682b7-5355-4df1-96dd-d558c475c7a7-kube-api-access-bcfsq\") pod \"authorino-c548b5d7d-ssjvm\" (UID: \"c81682b7-5355-4df1-96dd-d558c475c7a7\") " pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:36:53.146955 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:53.146920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:36:53.212681 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:53.212645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" event={"ID":"2d755fba-ee33-4897-a260-90a53b5ed7c4","Type":"ContainerStarted","Data":"e40a3aff9f3ac0cb64adc59c3bd81b23ef89c50e78738d29eba7a7aba792d044"} Aug 04 12:36:53.303408 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:53.303381 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ssjvm"] Aug 04 12:36:53.306706 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:36:53.306674 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc81682b7_5355_4df1_96dd_d558c475c7a7.slice/crio-113b3608c085b4483a9a3d5668a0c5dbaf4c006eb31a9da5463f5032bb3e6c20 WatchSource:0}: Error finding container 113b3608c085b4483a9a3d5668a0c5dbaf4c006eb31a9da5463f5032bb3e6c20: Status 404 returned error can't find the container with id 113b3608c085b4483a9a3d5668a0c5dbaf4c006eb31a9da5463f5032bb3e6c20 Aug 04 12:36:54.219139 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:54.219084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" event={"ID":"c81682b7-5355-4df1-96dd-d558c475c7a7","Type":"ContainerStarted","Data":"113b3608c085b4483a9a3d5668a0c5dbaf4c006eb31a9da5463f5032bb3e6c20"} Aug 04 12:36:55.226513 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:55.226474 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" event={"ID":"2d755fba-ee33-4897-a260-90a53b5ed7c4","Type":"ContainerStarted","Data":"dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842"} Aug 04 12:36:55.227016 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:55.226657 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:36:55.248768 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:55.248716 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" podStartSLOduration=2.055238744 podStartE2EDuration="4.248699468s" totalImagesPullingTime="2.193460724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:36:55.247265071 +0000 UTC m=+546.202860863" watchObservedRunningTime="2026-08-04 12:36:55.248699468 +0000 UTC m=+546.204295262" Aug 04 12:36:56.232585 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:56.232541 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" event={"ID":"c81682b7-5355-4df1-96dd-d558c475c7a7","Type":"ContainerStarted","Data":"f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0"} Aug 04 12:36:56.253114 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:36:56.253067 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" podStartSLOduration=2.028474939 podStartE2EDuration="4.253051693s" totalImagesPullingTime="2.224576754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:36:56.251445269 +0000 UTC m=+547.207041064" watchObservedRunningTime="2026-08-04 12:36:56.253051693 +0000 UTC m=+547.208647487" Aug 04 12:37:06.233968 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:06.233937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:37:07.020768 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.020736 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:37:07.020978 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.020952 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" podUID="2d755fba-ee33-4897-a260-90a53b5ed7c4" containerName="limitador" containerID="cri-o://dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842" gracePeriod=30 Aug 04 12:37:07.575236 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.575202 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:37:07.691209 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.691176 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d755fba-ee33-4897-a260-90a53b5ed7c4-config-file\") pod \"2d755fba-ee33-4897-a260-90a53b5ed7c4\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " Aug 04 12:37:07.691414 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.691242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr9zq\" (UniqueName: \"kubernetes.io/projected/2d755fba-ee33-4897-a260-90a53b5ed7c4-kube-api-access-nr9zq\") pod \"2d755fba-ee33-4897-a260-90a53b5ed7c4\" (UID: \"2d755fba-ee33-4897-a260-90a53b5ed7c4\") " Aug 04 12:37:07.691610 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.691583 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d755fba-ee33-4897-a260-90a53b5ed7c4-config-file" (OuterVolumeSpecName: "config-file") pod "2d755fba-ee33-4897-a260-90a53b5ed7c4" (UID: "2d755fba-ee33-4897-a260-90a53b5ed7c4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:37:07.693544 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.693521 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d755fba-ee33-4897-a260-90a53b5ed7c4-kube-api-access-nr9zq" (OuterVolumeSpecName: "kube-api-access-nr9zq") pod "2d755fba-ee33-4897-a260-90a53b5ed7c4" (UID: "2d755fba-ee33-4897-a260-90a53b5ed7c4"). InnerVolumeSpecName "kube-api-access-nr9zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:37:07.791862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.791830 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d755fba-ee33-4897-a260-90a53b5ed7c4-config-file\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:07.791862 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:07.791863 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9zq\" (UniqueName: \"kubernetes.io/projected/2d755fba-ee33-4897-a260-90a53b5ed7c4-kube-api-access-nr9zq\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:08.289351 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.289310 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d755fba-ee33-4897-a260-90a53b5ed7c4" containerID="dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842" exitCode=0 Aug 04 12:37:08.289351 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.289354 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" Aug 04 12:37:08.289594 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.289402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" event={"ID":"2d755fba-ee33-4897-a260-90a53b5ed7c4","Type":"ContainerDied","Data":"dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842"} Aug 04 12:37:08.289594 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.289453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-7j8rp" event={"ID":"2d755fba-ee33-4897-a260-90a53b5ed7c4","Type":"ContainerDied","Data":"e40a3aff9f3ac0cb64adc59c3bd81b23ef89c50e78738d29eba7a7aba792d044"} Aug 04 12:37:08.289594 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.289488 2578 scope.go:117] "RemoveContainer" containerID="dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842" Aug 04 12:37:08.298199 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.298183 2578 scope.go:117] "RemoveContainer" containerID="dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842" Aug 04 12:37:08.298467 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:08.298448 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842\": container with ID starting with dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842 not found: ID does not exist" containerID="dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842" Aug 04 12:37:08.298521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.298476 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842"} err="failed to get container status \"dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842\": rpc error: code = NotFound desc = could not find container \"dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842\": container with ID starting with dfd726587c895b8faf214dc09f33e2e20e981599d6c6d09610b5b0d06b53a842 not found: ID does not exist" Aug 04 12:37:08.312088 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.312060 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:37:08.316564 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:08.316544 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-7j8rp"] Aug 04 12:37:09.553228 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:09.553193 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d755fba-ee33-4897-a260-90a53b5ed7c4" path="/var/lib/kubelet/pods/2d755fba-ee33-4897-a260-90a53b5ed7c4/volumes" Aug 04 12:37:12.797878 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:12.797847 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:12.858824 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:12.858794 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:13.213405 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:13.213374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:15.135085 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:15.135048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:15.196500 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:15.196474 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:25.802336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:25.802296 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:26.128400 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.128365 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:26.217868 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.217829 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-clc5k"] Aug 04 12:37:26.218329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.218312 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d755fba-ee33-4897-a260-90a53b5ed7c4" containerName="limitador" Aug 04 12:37:26.218329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.218331 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d755fba-ee33-4897-a260-90a53b5ed7c4" containerName="limitador" Aug 04 12:37:26.218495 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.218412 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d755fba-ee33-4897-a260-90a53b5ed7c4" containerName="limitador" Aug 04 12:37:26.221533 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.221513 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.224219 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.224201 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 12:37:26.230092 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.230069 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-clc5k"] Aug 04 12:37:26.280169 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.280130 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-clc5k"] Aug 04 12:37:26.280425 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:26.280402 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mfkdn tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-mfkdn tls-cert]: context canceled" pod="kuadrant-system/authorino-687bd45585-clc5k" podUID="0af2bef2-3c03-4de0-9b9a-fede07aaa66b" Aug 04 12:37:26.310474 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.310438 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d6f5ff99-qg6g5"] Aug 04 12:37:26.313842 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.313824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.324166 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.324134 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d6f5ff99-qg6g5"] Aug 04 12:37:26.356105 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.356069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-tls-cert\") pod \"authorino-687bd45585-clc5k\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.356255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.356213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkdn\" (UniqueName: \"kubernetes.io/projected/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-kube-api-access-mfkdn\") pod \"authorino-687bd45585-clc5k\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.372727 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.372694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.379338 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.379287 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.456979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.456946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkdn\" (UniqueName: \"kubernetes.io/projected/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-kube-api-access-mfkdn\") pod \"authorino-687bd45585-clc5k\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.457208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.456985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfjw\" (UniqueName: \"kubernetes.io/projected/9cab951c-a22f-4555-8557-4d01293bed43-kube-api-access-lsfjw\") pod \"authorino-7d6f5ff99-qg6g5\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.457208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.457025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-tls-cert\") pod \"authorino-687bd45585-clc5k\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.457208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.457055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cab951c-a22f-4555-8557-4d01293bed43-tls-cert\") pod \"authorino-7d6f5ff99-qg6g5\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.459678 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.459648 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-tls-cert\") pod \"authorino-687bd45585-clc5k\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.467526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.467501 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkdn\" (UniqueName: \"kubernetes.io/projected/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-kube-api-access-mfkdn\") pod \"authorino-687bd45585-clc5k\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:26.557518 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.557484 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfkdn\" (UniqueName: \"kubernetes.io/projected/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-kube-api-access-mfkdn\") pod \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " Aug 04 12:37:26.557696 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.557570 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-tls-cert\") pod \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\" (UID: \"0af2bef2-3c03-4de0-9b9a-fede07aaa66b\") " Aug 04 12:37:26.557696 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.557680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cab951c-a22f-4555-8557-4d01293bed43-tls-cert\") pod \"authorino-7d6f5ff99-qg6g5\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.557780 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.557731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfjw\" (UniqueName: \"kubernetes.io/projected/9cab951c-a22f-4555-8557-4d01293bed43-kube-api-access-lsfjw\") pod \"authorino-7d6f5ff99-qg6g5\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.559829 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.559789 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-kube-api-access-mfkdn" (OuterVolumeSpecName: "kube-api-access-mfkdn") pod "0af2bef2-3c03-4de0-9b9a-fede07aaa66b" (UID: "0af2bef2-3c03-4de0-9b9a-fede07aaa66b"). InnerVolumeSpecName "kube-api-access-mfkdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:37:26.559829 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.559796 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0af2bef2-3c03-4de0-9b9a-fede07aaa66b" (UID: "0af2bef2-3c03-4de0-9b9a-fede07aaa66b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:37:26.560215 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.560197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cab951c-a22f-4555-8557-4d01293bed43-tls-cert\") pod \"authorino-7d6f5ff99-qg6g5\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.568204 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.568174 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfjw\" (UniqueName: \"kubernetes.io/projected/9cab951c-a22f-4555-8557-4d01293bed43-kube-api-access-lsfjw\") pod \"authorino-7d6f5ff99-qg6g5\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.629947 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.629855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:37:26.658493 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.658466 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfkdn\" (UniqueName: \"kubernetes.io/projected/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-kube-api-access-mfkdn\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:26.658493 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.658492 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0af2bef2-3c03-4de0-9b9a-fede07aaa66b-tls-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:26.765624 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:26.765600 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d6f5ff99-qg6g5"] Aug 04 12:37:26.767625 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:37:26.767599 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cab951c_a22f_4555_8557_4d01293bed43.slice/crio-be5d9e849049da25a646c2d96a6d95bcbc7f89ee27ddfcb4bb753fd3411c02f2 WatchSource:0}: Error finding container be5d9e849049da25a646c2d96a6d95bcbc7f89ee27ddfcb4bb753fd3411c02f2: Status 404 returned error can't find the container with id be5d9e849049da25a646c2d96a6d95bcbc7f89ee27ddfcb4bb753fd3411c02f2 Aug 04 12:37:27.379056 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:27.379025 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-clc5k" Aug 04 12:37:27.379056 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:27.379047 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" event={"ID":"9cab951c-a22f-4555-8557-4d01293bed43","Type":"ContainerStarted","Data":"be5d9e849049da25a646c2d96a6d95bcbc7f89ee27ddfcb4bb753fd3411c02f2"} Aug 04 12:37:27.416369 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:27.416340 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-clc5k"] Aug 04 12:37:27.425186 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:27.425151 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-clc5k"] Aug 04 12:37:27.553790 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:27.553757 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af2bef2-3c03-4de0-9b9a-fede07aaa66b" path="/var/lib/kubelet/pods/0af2bef2-3c03-4de0-9b9a-fede07aaa66b/volumes" Aug 04 12:37:28.385986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.385953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" event={"ID":"9cab951c-a22f-4555-8557-4d01293bed43","Type":"ContainerStarted","Data":"0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129"} Aug 04 12:37:28.406640 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.406590 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" podStartSLOduration=1.8336718730000001 podStartE2EDuration="2.40657364s" totalImagesPullingTime="572.901767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:37:28.405056276 +0000 UTC m=+579.360652070" watchObservedRunningTime="2026-08-04 12:37:28.40657364 +0000 UTC m=+579.362169436" Aug 04 12:37:28.438554 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.438519 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ssjvm"] Aug 04 12:37:28.438768 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.438742 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" podUID="c81682b7-5355-4df1-96dd-d558c475c7a7" containerName="authorino" containerID="cri-o://f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0" gracePeriod=30 Aug 04 12:37:28.716976 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.716945 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:37:28.878378 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.878338 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcfsq\" (UniqueName: \"kubernetes.io/projected/c81682b7-5355-4df1-96dd-d558c475c7a7-kube-api-access-bcfsq\") pod \"c81682b7-5355-4df1-96dd-d558c475c7a7\" (UID: \"c81682b7-5355-4df1-96dd-d558c475c7a7\") " Aug 04 12:37:28.880504 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.880474 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81682b7-5355-4df1-96dd-d558c475c7a7-kube-api-access-bcfsq" (OuterVolumeSpecName: "kube-api-access-bcfsq") pod "c81682b7-5355-4df1-96dd-d558c475c7a7" (UID: "c81682b7-5355-4df1-96dd-d558c475c7a7"). InnerVolumeSpecName "kube-api-access-bcfsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:37:28.979178 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:28.979057 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcfsq\" (UniqueName: \"kubernetes.io/projected/c81682b7-5355-4df1-96dd-d558c475c7a7-kube-api-access-bcfsq\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:29.391834 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.391797 2578 generic.go:356] "Generic (PLEG): container finished" podID="c81682b7-5355-4df1-96dd-d558c475c7a7" containerID="f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0" exitCode=0 Aug 04 12:37:29.391834 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.391837 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" Aug 04 12:37:29.392457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.391882 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" event={"ID":"c81682b7-5355-4df1-96dd-d558c475c7a7","Type":"ContainerDied","Data":"f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0"} Aug 04 12:37:29.392457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.391925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-ssjvm" event={"ID":"c81682b7-5355-4df1-96dd-d558c475c7a7","Type":"ContainerDied","Data":"113b3608c085b4483a9a3d5668a0c5dbaf4c006eb31a9da5463f5032bb3e6c20"} Aug 04 12:37:29.392457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.391951 2578 scope.go:117] "RemoveContainer" containerID="f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0" Aug 04 12:37:29.401297 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.401280 2578 scope.go:117] "RemoveContainer" containerID="f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0" Aug 04 12:37:29.401568 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:29.401551 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0\": container with ID starting with f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0 not found: ID does not exist" containerID="f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0" Aug 04 12:37:29.401619 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.401577 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0"} err="failed to get container status \"f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0\": rpc error: code = NotFound desc = could not find container \"f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0\": container with ID starting with f4a45be7ffc7f8dbfd36c3f6d00557a4eb08a39024f05008c42011254687ebf0 not found: ID does not exist" Aug 04 12:37:29.414650 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.414622 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ssjvm"] Aug 04 12:37:29.419601 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.419576 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-ssjvm"] Aug 04 12:37:29.552835 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:29.552800 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81682b7-5355-4df1-96dd-d558c475c7a7" path="/var/lib/kubelet/pods/c81682b7-5355-4df1-96dd-d558c475c7a7/volumes" Aug 04 12:37:32.904077 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.904046 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:32.907927 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.907896 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cfbc67dd-976sg"] Aug 04 12:37:32.908342 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.908324 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81682b7-5355-4df1-96dd-d558c475c7a7" containerName="authorino" Aug 04 12:37:32.908453 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.908344 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81682b7-5355-4df1-96dd-d558c475c7a7" containerName="authorino" Aug 04 12:37:32.908527 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.908455 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81682b7-5355-4df1-96dd-d558c475c7a7" containerName="authorino" Aug 04 12:37:32.911111 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.911093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:32.913983 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.913962 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 12:37:32.914094 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.913963 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cw2b9\"" Aug 04 12:37:32.914094 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.913966 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 12:37:32.922683 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:32.922665 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cfbc67dd-976sg"] Aug 04 12:37:33.004815 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.004782 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:33.016979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.016954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbv6s\" (UniqueName: \"kubernetes.io/projected/abbd6742-e409-4d78-9135-b62317bde485-kube-api-access-nbv6s\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.017141 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.016988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.079633 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.079600 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:33.083843 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.083814 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d8768cdd-drqk6"] Aug 04 12:37:33.086083 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.086068 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.098428 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.098401 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-drqk6"] Aug 04 12:37:33.117989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.117962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbv6s\" (UniqueName: \"kubernetes.io/projected/abbd6742-e409-4d78-9135-b62317bde485-kube-api-access-nbv6s\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.118178 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.117998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.118258 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:33.118191 2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 12:37:33.118315 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:33.118260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert podName:abbd6742-e409-4d78-9135-b62317bde485 nodeName:}" failed. No retries permitted until 2026-08-04 12:37:33.618237137 +0000 UTC m=+584.573832924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert") pod "maas-controller-5cfbc67dd-976sg" (UID: "abbd6742-e409-4d78-9135-b62317bde485") : secret "maas-controller-webhook-cert" not found Aug 04 12:37:33.129849 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.129820 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbv6s\" (UniqueName: \"kubernetes.io/projected/abbd6742-e409-4d78-9135-b62317bde485-kube-api-access-nbv6s\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.219300 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.219211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.219448 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.219329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sk9q\" (UniqueName: \"kubernetes.io/projected/841dbb8a-f049-4e7a-ba52-968932a44c60-kube-api-access-5sk9q\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.320235 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.320202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.320381 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.320311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sk9q\" (UniqueName: \"kubernetes.io/projected/841dbb8a-f049-4e7a-ba52-968932a44c60-kube-api-access-5sk9q\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.320381 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:33.320353 2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 12:37:33.320461 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:33.320422 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert podName:841dbb8a-f049-4e7a-ba52-968932a44c60 nodeName:}" failed. No retries permitted until 2026-08-04 12:37:33.820405917 +0000 UTC m=+584.776001691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert") pod "maas-controller-68d8768cdd-drqk6" (UID: "841dbb8a-f049-4e7a-ba52-968932a44c60") : secret "maas-controller-webhook-cert" not found Aug 04 12:37:33.331861 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.331838 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sk9q\" (UniqueName: \"kubernetes.io/projected/841dbb8a-f049-4e7a-ba52-968932a44c60-kube-api-access-5sk9q\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.415934 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.415901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:33.622974 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.622942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.625609 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.625578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert\") pod \"maas-controller-5cfbc67dd-976sg\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.822237 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.822201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:33.824693 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.824275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.826755 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.826728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert\") pod \"maas-controller-68d8768cdd-drqk6\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:33.958820 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.958796 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cfbc67dd-976sg"] Aug 04 12:37:33.961098 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:37:33.961073 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbd6742_e409_4d78_9135_b62317bde485.slice/crio-1cc92f3f46296469026e0c696bc17cf0605f2c1fd517f8f3ecc443806a59a86d WatchSource:0}: Error finding container 1cc92f3f46296469026e0c696bc17cf0605f2c1fd517f8f3ecc443806a59a86d: Status 404 returned error can't find the container with id 1cc92f3f46296469026e0c696bc17cf0605f2c1fd517f8f3ecc443806a59a86d Aug 04 12:37:33.996503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:33.996472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:34.128153 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:34.128062 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-drqk6"] Aug 04 12:37:34.131472 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:37:34.131436 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841dbb8a_f049_4e7a_ba52_968932a44c60.slice/crio-9a3d569d69f821d4a9130225add5bc86ccc35c943cf91050348df63e111d2f10 WatchSource:0}: Error finding container 9a3d569d69f821d4a9130225add5bc86ccc35c943cf91050348df63e111d2f10: Status 404 returned error can't find the container with id 9a3d569d69f821d4a9130225add5bc86ccc35c943cf91050348df63e111d2f10 Aug 04 12:37:34.422422 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:34.422323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-drqk6" event={"ID":"841dbb8a-f049-4e7a-ba52-968932a44c60","Type":"ContainerStarted","Data":"9a3d569d69f821d4a9130225add5bc86ccc35c943cf91050348df63e111d2f10"} Aug 04 12:37:34.423506 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:34.423475 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfbc67dd-976sg" event={"ID":"abbd6742-e409-4d78-9135-b62317bde485","Type":"ContainerStarted","Data":"1cc92f3f46296469026e0c696bc17cf0605f2c1fd517f8f3ecc443806a59a86d"} Aug 04 12:37:36.439694 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:36.439655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfbc67dd-976sg" event={"ID":"abbd6742-e409-4d78-9135-b62317bde485","Type":"ContainerStarted","Data":"442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090"} Aug 04 12:37:36.440073 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:36.439735 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:36.464621 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:36.464564 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cfbc67dd-976sg" podStartSLOduration=2.118375789 podStartE2EDuration="4.464549246s" totalImagesPullingTime="2.346173457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:37:36.463702151 +0000 UTC m=+587.419297946" watchObservedRunningTime="2026-08-04 12:37:36.464549246 +0000 UTC m=+587.420145039" Aug 04 12:37:36.476433 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:36.476405 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:37.445477 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:37.445444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-drqk6" event={"ID":"841dbb8a-f049-4e7a-ba52-968932a44c60","Type":"ContainerStarted","Data":"cc08cfcca35f69754d9741f1c618cb81fa5db1a7ae5225a83340067cd5074155"} Aug 04 12:37:37.445899 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:37.445691 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:37.467168 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:37.467093 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d8768cdd-drqk6" podStartSLOduration=2.100999063 podStartE2EDuration="4.467076214s" totalImagesPullingTime="2.366077151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:37:37.46479111 +0000 UTC m=+588.420386903" watchObservedRunningTime="2026-08-04 12:37:37.467076214 +0000 UTC m=+588.422672008" Aug 04 12:37:38.092673 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:38.092636 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:47.448044 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:47.448011 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:48.452459 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.452429 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:37:48.511645 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.511617 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cfbc67dd-976sg"] Aug 04 12:37:48.511823 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.511801 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cfbc67dd-976sg" podUID="abbd6742-e409-4d78-9135-b62317bde485" containerName="manager" containerID="cri-o://442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090" gracePeriod=30 Aug 04 12:37:48.760691 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.760661 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:48.864782 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.864747 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert\") pod \"abbd6742-e409-4d78-9135-b62317bde485\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " Aug 04 12:37:48.864782 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.864787 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbv6s\" (UniqueName: \"kubernetes.io/projected/abbd6742-e409-4d78-9135-b62317bde485-kube-api-access-nbv6s\") pod \"abbd6742-e409-4d78-9135-b62317bde485\" (UID: \"abbd6742-e409-4d78-9135-b62317bde485\") " Aug 04 12:37:48.867222 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.867182 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "abbd6742-e409-4d78-9135-b62317bde485" (UID: "abbd6742-e409-4d78-9135-b62317bde485"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:37:48.867340 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.867268 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd6742-e409-4d78-9135-b62317bde485-kube-api-access-nbv6s" (OuterVolumeSpecName: "kube-api-access-nbv6s") pod "abbd6742-e409-4d78-9135-b62317bde485" (UID: "abbd6742-e409-4d78-9135-b62317bde485"). InnerVolumeSpecName "kube-api-access-nbv6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:37:48.965576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.965541 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abbd6742-e409-4d78-9135-b62317bde485-webhook-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:48.965576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:48.965570 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbv6s\" (UniqueName: \"kubernetes.io/projected/abbd6742-e409-4d78-9135-b62317bde485-kube-api-access-nbv6s\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:37:49.423442 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.423414 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:37:49.423616 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.423414 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:37:49.503918 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.503880 2578 generic.go:356] "Generic (PLEG): container finished" podID="abbd6742-e409-4d78-9135-b62317bde485" containerID="442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090" exitCode=0 Aug 04 12:37:49.504332 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.503926 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfbc67dd-976sg" Aug 04 12:37:49.504332 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.503965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfbc67dd-976sg" event={"ID":"abbd6742-e409-4d78-9135-b62317bde485","Type":"ContainerDied","Data":"442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090"} Aug 04 12:37:49.504332 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.504018 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfbc67dd-976sg" event={"ID":"abbd6742-e409-4d78-9135-b62317bde485","Type":"ContainerDied","Data":"1cc92f3f46296469026e0c696bc17cf0605f2c1fd517f8f3ecc443806a59a86d"} Aug 04 12:37:49.504332 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.504043 2578 scope.go:117] "RemoveContainer" containerID="442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090" Aug 04 12:37:49.512936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.512755 2578 scope.go:117] "RemoveContainer" containerID="442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090" Aug 04 12:37:49.513059 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:37:49.513040 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090\": container with ID starting with 442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090 not found: ID does not exist" containerID="442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090" Aug 04 12:37:49.513103 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.513069 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090"} err="failed to get container status \"442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090\": rpc error: code = NotFound desc = could not find container \"442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090\": container with ID starting with 442212d39aa32a330cf6597c7bf196483dbe7104c8389a6929a93b8c2811f090 not found: ID does not exist" Aug 04 12:37:49.527306 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.527282 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cfbc67dd-976sg"] Aug 04 12:37:49.533603 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.533582 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cfbc67dd-976sg"] Aug 04 12:37:49.552810 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:49.552782 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abbd6742-e409-4d78-9135-b62317bde485" path="/var/lib/kubelet/pods/abbd6742-e409-4d78-9135-b62317bde485/volumes" Aug 04 12:37:50.767237 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:50.767200 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:37:52.082539 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:37:52.082504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:07.905490 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:07.905451 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:08.006168 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:08.006111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:09.460852 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.460820 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:09.718890 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.718814 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb"] Aug 04 12:38:09.719196 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.719184 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbd6742-e409-4d78-9135-b62317bde485" containerName="manager" Aug 04 12:38:09.719249 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.719197 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbd6742-e409-4d78-9135-b62317bde485" containerName="manager" Aug 04 12:38:09.719285 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.719258 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbd6742-e409-4d78-9135-b62317bde485" containerName="manager" Aug 04 12:38:09.722457 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.722437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.725700 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.725682 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 12:38:09.725840 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.725689 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 12:38:09.725840 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.725745 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 12:38:09.726953 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.726939 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p76m5\"" Aug 04 12:38:09.733448 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.733428 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb"] Aug 04 12:38:09.861799 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.861764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd5b390-56a1-4127-b4fd-7631f66c54bc-openshift-service-ca\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.861799 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.861800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlp8\" (UniqueName: \"kubernetes.io/projected/4fd5b390-56a1-4127-b4fd-7631f66c54bc-kube-api-access-mhlp8\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.862012 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.861960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd5b390-56a1-4127-b4fd-7631f66c54bc-maas-api-tls\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.962765 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.962684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd5b390-56a1-4127-b4fd-7631f66c54bc-openshift-service-ca\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.962765 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.962725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlp8\" (UniqueName: \"kubernetes.io/projected/4fd5b390-56a1-4127-b4fd-7631f66c54bc-kube-api-access-mhlp8\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.962945 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.962808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd5b390-56a1-4127-b4fd-7631f66c54bc-maas-api-tls\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.962945 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:38:09.962910 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Aug 04 12:38:09.963015 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:38:09.962983 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fd5b390-56a1-4127-b4fd-7631f66c54bc-maas-api-tls podName:4fd5b390-56a1-4127-b4fd-7631f66c54bc nodeName:}" failed. No retries permitted until 2026-08-04 12:38:10.462963262 +0000 UTC m=+621.418559034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fd5b390-56a1-4127-b4fd-7631f66c54bc-maas-api-tls") pod "maas-api-8544796ddc-h6bqb" (UID: "4fd5b390-56a1-4127-b4fd-7631f66c54bc") : secret "maas-api-serving-cert" not found Aug 04 12:38:09.963356 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.963337 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd5b390-56a1-4127-b4fd-7631f66c54bc-openshift-service-ca\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:09.972802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:09.972744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlp8\" (UniqueName: \"kubernetes.io/projected/4fd5b390-56a1-4127-b4fd-7631f66c54bc-kube-api-access-mhlp8\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:10.075815 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.075784 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:10.182179 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.182108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:10.221267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.221234 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wjhrj"] Aug 04 12:38:10.228000 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.227943 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.231015 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.230995 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 12:38:10.231195 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.231177 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sz2vr\"" Aug 04 12:38:10.234972 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.234933 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wjhrj"] Aug 04 12:38:10.252478 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.252452 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:10.366610 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.366572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34420b0a-98d8-4359-b89d-574811fe369d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wjhrj\" (UID: \"34420b0a-98d8-4359-b89d-574811fe369d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.366792 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.366717 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzhh\" (UniqueName: \"kubernetes.io/projected/34420b0a-98d8-4359-b89d-574811fe369d-kube-api-access-lnzhh\") pod \"payload-processing-5cc6cd444b-wjhrj\" (UID: \"34420b0a-98d8-4359-b89d-574811fe369d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.468164 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.468103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34420b0a-98d8-4359-b89d-574811fe369d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wjhrj\" (UID: \"34420b0a-98d8-4359-b89d-574811fe369d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.468632 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.468212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd5b390-56a1-4127-b4fd-7631f66c54bc-maas-api-tls\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:10.468632 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.468232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzhh\" (UniqueName: \"kubernetes.io/projected/34420b0a-98d8-4359-b89d-574811fe369d-kube-api-access-lnzhh\") pod \"payload-processing-5cc6cd444b-wjhrj\" (UID: \"34420b0a-98d8-4359-b89d-574811fe369d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.468867 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.468846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34420b0a-98d8-4359-b89d-574811fe369d-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-wjhrj\" (UID: \"34420b0a-98d8-4359-b89d-574811fe369d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.470827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.470803 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fd5b390-56a1-4127-b4fd-7631f66c54bc-maas-api-tls\") pod \"maas-api-8544796ddc-h6bqb\" (UID: \"4fd5b390-56a1-4127-b4fd-7631f66c54bc\") " pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:10.477985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.477963 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzhh\" (UniqueName: \"kubernetes.io/projected/34420b0a-98d8-4359-b89d-574811fe369d-kube-api-access-lnzhh\") pod \"payload-processing-5cc6cd444b-wjhrj\" (UID: \"34420b0a-98d8-4359-b89d-574811fe369d\") " pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.541053 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.540957 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:10.633283 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.633248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:10.695656 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.695623 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-wjhrj"] Aug 04 12:38:10.698702 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:38:10.698665 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34420b0a_98d8_4359_b89d_574811fe369d.slice/crio-d38cfc5b75b70b4bfd4eec2f73669a82b53b41364621114b4dfb10d1bda324a9 WatchSource:0}: Error finding container d38cfc5b75b70b4bfd4eec2f73669a82b53b41364621114b4dfb10d1bda324a9: Status 404 returned error can't find the container with id d38cfc5b75b70b4bfd4eec2f73669a82b53b41364621114b4dfb10d1bda324a9 Aug 04 12:38:10.781997 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:10.781962 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb"] Aug 04 12:38:10.786439 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:38:10.786409 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd5b390_56a1_4127_b4fd_7631f66c54bc.slice/crio-43d58ca0de86080790408b963aaeca69f7dbeea0d38aaa0e44bcc8d7f6e32f22 WatchSource:0}: Error finding container 43d58ca0de86080790408b963aaeca69f7dbeea0d38aaa0e44bcc8d7f6e32f22: Status 404 returned error can't find the container with id 43d58ca0de86080790408b963aaeca69f7dbeea0d38aaa0e44bcc8d7f6e32f22 Aug 04 12:38:11.614587 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:11.614499 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" event={"ID":"4fd5b390-56a1-4127-b4fd-7631f66c54bc","Type":"ContainerStarted","Data":"43d58ca0de86080790408b963aaeca69f7dbeea0d38aaa0e44bcc8d7f6e32f22"} Aug 04 12:38:11.619219 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:11.619141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" event={"ID":"34420b0a-98d8-4359-b89d-574811fe369d","Type":"ContainerStarted","Data":"d38cfc5b75b70b4bfd4eec2f73669a82b53b41364621114b4dfb10d1bda324a9"} Aug 04 12:38:13.631530 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:13.631485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" event={"ID":"34420b0a-98d8-4359-b89d-574811fe369d","Type":"ContainerStarted","Data":"973fe9c185d85b13fb0e3c5168dc3b545d36ef5f8eee75ad0cc7ed0fe1b85293"} Aug 04 12:38:13.632011 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:13.631669 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:13.632994 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:13.632971 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" event={"ID":"4fd5b390-56a1-4127-b4fd-7631f66c54bc","Type":"ContainerStarted","Data":"25cbc98beb1beb2fc7517d4f6c6b7a22cd9ae8b79ce3b1b407114a81d6e2bbaf"} Aug 04 12:38:13.633136 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:13.633029 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:13.651111 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:13.651069 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" podStartSLOduration=1.6143881690000002 podStartE2EDuration="3.651055444s" totalImagesPullingTime="2.036667275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:38:13.649098764 +0000 UTC m=+624.604694559" watchObservedRunningTime="2026-08-04 12:38:13.651055444 +0000 UTC m=+624.606651216" Aug 04 12:38:13.669397 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:13.669352 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" podStartSLOduration=2.726054718 podStartE2EDuration="4.669336017s" totalImagesPullingTime="1.943281299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:38:13.667721277 +0000 UTC m=+624.623317065" watchObservedRunningTime="2026-08-04 12:38:13.669336017 +0000 UTC m=+624.624931811" Aug 04 12:38:19.644583 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:19.644555 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8544796ddc-h6bqb" Aug 04 12:38:20.618779 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:20.618751 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:20.717030 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:20.716991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:24.640767 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.640738 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-wjhrj" Aug 04 12:38:24.824790 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.824758 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6"] Aug 04 12:38:24.828022 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.827993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.831191 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.831105 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 12:38:24.831446 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.831427 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mddbc\"" Aug 04 12:38:24.831532 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.831467 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 04 12:38:24.831600 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.831435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 12:38:24.838582 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.838558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6"] Aug 04 12:38:24.849880 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.849854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:24.890538 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.890505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388c7314-2222-425e-952b-47ca549d65fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.890695 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.890550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzcq\" (UniqueName: \"kubernetes.io/projected/388c7314-2222-425e-952b-47ca549d65fe-kube-api-access-bxzcq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.890695 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.890654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.890695 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.890680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.890879 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.890703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.890879 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.890731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991498 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991498 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991499 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991735 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991735 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991735 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388c7314-2222-425e-952b-47ca549d65fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991735 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzcq\" (UniqueName: \"kubernetes.io/projected/388c7314-2222-425e-952b-47ca549d65fe-kube-api-access-bxzcq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.991940 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991873 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.992002 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.991940 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.992066 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.992042 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.994053 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.994028 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/388c7314-2222-425e-952b-47ca549d65fe-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:24.994387 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:24.994370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/388c7314-2222-425e-952b-47ca549d65fe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:25.003077 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:25.003050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzcq\" (UniqueName: \"kubernetes.io/projected/388c7314-2222-425e-952b-47ca549d65fe-kube-api-access-bxzcq\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6\" (UID: \"388c7314-2222-425e-952b-47ca549d65fe\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:25.140316 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:25.140278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:25.275222 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:25.275196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6"] Aug 04 12:38:25.277048 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:38:25.277015 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388c7314_2222_425e_952b_47ca549d65fe.slice/crio-bfa7c797ff7f3157a3aa369baa2fbfc9f43db5b276b233909af4baba43a5e230 WatchSource:0}: Error finding container bfa7c797ff7f3157a3aa369baa2fbfc9f43db5b276b233909af4baba43a5e230: Status 404 returned error can't find the container with id bfa7c797ff7f3157a3aa369baa2fbfc9f43db5b276b233909af4baba43a5e230 Aug 04 12:38:25.700202 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:25.700159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" event={"ID":"388c7314-2222-425e-952b-47ca549d65fe","Type":"ContainerStarted","Data":"bfa7c797ff7f3157a3aa369baa2fbfc9f43db5b276b233909af4baba43a5e230"} Aug 04 12:38:26.708241 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:26.708206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" event={"ID":"388c7314-2222-425e-952b-47ca549d65fe","Type":"ContainerStarted","Data":"69ea50f5819e63fcffcb6a0f860d674d2ef1be992395ef2a0719b5de641c3622"} Aug 04 12:38:26.729718 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:26.729665 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" podStartSLOduration=1.475742813 podStartE2EDuration="2.729646971s" totalImagesPullingTime="1.253904158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:38:26.728341484 +0000 UTC m=+637.683937291" watchObservedRunningTime="2026-08-04 12:38:26.729646971 +0000 UTC m=+637.685242764" Aug 04 12:38:31.786889 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.786858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7"] Aug 04 12:38:31.790606 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.790588 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.793321 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.793303 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Aug 04 12:38:31.800651 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.800630 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7"] Aug 04 12:38:31.823841 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.823811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:38:31.855944 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.855911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.856079 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.855948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.856079 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.855993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.856079 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.856038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28jt\" (UniqueName: \"kubernetes.io/projected/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-kube-api-access-d28jt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.856251 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.856087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.856251 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.856144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957048 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28jt\" (UniqueName: \"kubernetes.io/projected/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-kube-api-access-d28jt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957621 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957601 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957690 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957638 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.957690 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.957651 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.959554 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.959533 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.959679 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.959660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:31.966507 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:31.966484 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28jt\" (UniqueName: \"kubernetes.io/projected/d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e-kube-api-access-d28jt\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7bk7\" (UID: \"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:32.101388 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:32.101359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:32.236986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:32.236959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7"] Aug 04 12:38:32.239371 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:38:32.239345 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c4fb28_fc80_4bd7_90a4_a46ff68e7c0e.slice/crio-0892771c28e563048fab69585abb3722741a65b2a1fac295b1f53b1f69e84955 WatchSource:0}: Error finding container 0892771c28e563048fab69585abb3722741a65b2a1fac295b1f53b1f69e84955: Status 404 returned error can't find the container with id 0892771c28e563048fab69585abb3722741a65b2a1fac295b1f53b1f69e84955 Aug 04 12:38:32.241023 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:32.241004 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 12:38:32.740078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:32.739983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" event={"ID":"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e","Type":"ContainerStarted","Data":"c18269afafa0580b03a3acf7cf5ee997cb538c825b230a91d4e130f8c3e57146"} Aug 04 12:38:32.740078 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:32.740044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" event={"ID":"d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e","Type":"ContainerStarted","Data":"0892771c28e563048fab69585abb3722741a65b2a1fac295b1f53b1f69e84955"} Aug 04 12:38:32.760051 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:32.760002 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" podStartSLOduration=1.5588345810000002 podStartE2EDuration="1.759984613s" totalImagesPullingTime="201.150032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:38:32.759377083 +0000 UTC m=+643.714972911" watchObservedRunningTime="2026-08-04 12:38:32.759984613 +0000 UTC m=+643.715580411" Aug 04 12:38:35.140613 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:35.140575 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:35.141015 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:35.140671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:35.153346 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:35.153319 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:35.768707 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:35.768678 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6" Aug 04 12:38:42.102067 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:42.102019 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:42.102617 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:42.102088 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:42.114584 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:42.114560 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:38:42.802889 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:38:42.802862 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7bk7" Aug 04 12:39:05.618420 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:39:05.618386 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:39:17.927174 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:39:17.927114 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:39:21.515799 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:39:21.515768 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:39:38.340044 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:39:38.340006 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:39:57.394608 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:39:57.394570 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:40:18.811572 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:18.811529 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:40:20.985561 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:20.985531 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:40:21.086132 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:21.086089 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:40:56.523489 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:56.523427 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:40:56.583419 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:56.583384 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:40:59.641048 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.641016 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55d4964dc4-nq6lb"] Aug 04 12:40:59.644768 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.644748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.653630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.653602 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55d4964dc4-nq6lb"] Aug 04 12:40:59.836197 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.836164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/243410bc-221c-4bb5-8a52-6624a11c1f97-tls-cert\") pod \"authorino-55d4964dc4-nq6lb\" (UID: \"243410bc-221c-4bb5-8a52-6624a11c1f97\") " pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.836383 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.836308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsn45\" (UniqueName: \"kubernetes.io/projected/243410bc-221c-4bb5-8a52-6624a11c1f97-kube-api-access-nsn45\") pod \"authorino-55d4964dc4-nq6lb\" (UID: \"243410bc-221c-4bb5-8a52-6624a11c1f97\") " pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.936886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.936796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsn45\" (UniqueName: \"kubernetes.io/projected/243410bc-221c-4bb5-8a52-6624a11c1f97-kube-api-access-nsn45\") pod \"authorino-55d4964dc4-nq6lb\" (UID: \"243410bc-221c-4bb5-8a52-6624a11c1f97\") " pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.936886 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.936839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/243410bc-221c-4bb5-8a52-6624a11c1f97-tls-cert\") pod \"authorino-55d4964dc4-nq6lb\" (UID: \"243410bc-221c-4bb5-8a52-6624a11c1f97\") " pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.939448 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.939428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/243410bc-221c-4bb5-8a52-6624a11c1f97-tls-cert\") pod \"authorino-55d4964dc4-nq6lb\" (UID: \"243410bc-221c-4bb5-8a52-6624a11c1f97\") " pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.947966 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.947941 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsn45\" (UniqueName: \"kubernetes.io/projected/243410bc-221c-4bb5-8a52-6624a11c1f97-kube-api-access-nsn45\") pod \"authorino-55d4964dc4-nq6lb\" (UID: \"243410bc-221c-4bb5-8a52-6624a11c1f97\") " pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:40:59.954632 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:40:59.954598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55d4964dc4-nq6lb" Aug 04 12:41:00.094440 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:00.094412 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55d4964dc4-nq6lb"] Aug 04 12:41:00.096685 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:41:00.096658 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod243410bc_221c_4bb5_8a52_6624a11c1f97.slice/crio-b499a49955a329662c209500fe4b1d4f5ddbc8c9b08a35068a2212ae2adf4f10 WatchSource:0}: Error finding container b499a49955a329662c209500fe4b1d4f5ddbc8c9b08a35068a2212ae2adf4f10: Status 404 returned error can't find the container with id b499a49955a329662c209500fe4b1d4f5ddbc8c9b08a35068a2212ae2adf4f10 Aug 04 12:41:00.456336 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:00.456297 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55d4964dc4-nq6lb" event={"ID":"243410bc-221c-4bb5-8a52-6624a11c1f97","Type":"ContainerStarted","Data":"b499a49955a329662c209500fe4b1d4f5ddbc8c9b08a35068a2212ae2adf4f10"} Aug 04 12:41:01.464004 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.463956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55d4964dc4-nq6lb" event={"ID":"243410bc-221c-4bb5-8a52-6624a11c1f97","Type":"ContainerStarted","Data":"0cb15a8248880f3058ba9dd5f2b5f4d5af56c42478b19799a56f6b0e7b221fee"} Aug 04 12:41:01.555368 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.555305 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55d4964dc4-nq6lb" podStartSLOduration=1.907879816 podStartE2EDuration="2.55528169s" totalImagesPullingTime="647.401874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:41:01.48576686 +0000 UTC m=+792.441362654" watchObservedRunningTime="2026-08-04 12:41:01.55528169 +0000 UTC m=+792.510877485" Aug 04 12:41:01.559191 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.559160 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d6f5ff99-qg6g5"] Aug 04 12:41:01.559414 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.559387 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" podUID="9cab951c-a22f-4555-8557-4d01293bed43" containerName="authorino" containerID="cri-o://0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129" gracePeriod=30 Aug 04 12:41:01.822226 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.822201 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:41:01.957315 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.957276 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsfjw\" (UniqueName: \"kubernetes.io/projected/9cab951c-a22f-4555-8557-4d01293bed43-kube-api-access-lsfjw\") pod \"9cab951c-a22f-4555-8557-4d01293bed43\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " Aug 04 12:41:01.957503 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.957375 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cab951c-a22f-4555-8557-4d01293bed43-tls-cert\") pod \"9cab951c-a22f-4555-8557-4d01293bed43\" (UID: \"9cab951c-a22f-4555-8557-4d01293bed43\") " Aug 04 12:41:01.959576 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.959542 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cab951c-a22f-4555-8557-4d01293bed43-kube-api-access-lsfjw" (OuterVolumeSpecName: "kube-api-access-lsfjw") pod "9cab951c-a22f-4555-8557-4d01293bed43" (UID: "9cab951c-a22f-4555-8557-4d01293bed43"). InnerVolumeSpecName "kube-api-access-lsfjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:41:01.968615 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:01.968590 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cab951c-a22f-4555-8557-4d01293bed43-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9cab951c-a22f-4555-8557-4d01293bed43" (UID: "9cab951c-a22f-4555-8557-4d01293bed43"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:41:02.058683 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.058598 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfjw\" (UniqueName: \"kubernetes.io/projected/9cab951c-a22f-4555-8557-4d01293bed43-kube-api-access-lsfjw\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:41:02.058683 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.058643 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cab951c-a22f-4555-8557-4d01293bed43-tls-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:41:02.065974 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.065947 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:41:02.159109 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.159062 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:41:02.471497 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.471458 2578 generic.go:356] "Generic (PLEG): container finished" podID="9cab951c-a22f-4555-8557-4d01293bed43" containerID="0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129" exitCode=0 Aug 04 12:41:02.471982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.471526 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" Aug 04 12:41:02.471982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.471546 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" event={"ID":"9cab951c-a22f-4555-8557-4d01293bed43","Type":"ContainerDied","Data":"0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129"} Aug 04 12:41:02.471982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.471604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d6f5ff99-qg6g5" event={"ID":"9cab951c-a22f-4555-8557-4d01293bed43","Type":"ContainerDied","Data":"be5d9e849049da25a646c2d96a6d95bcbc7f89ee27ddfcb4bb753fd3411c02f2"} Aug 04 12:41:02.471982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.471628 2578 scope.go:117] "RemoveContainer" containerID="0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129" Aug 04 12:41:02.482049 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.482030 2578 scope.go:117] "RemoveContainer" containerID="0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129" Aug 04 12:41:02.482361 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:41:02.482343 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129\": container with ID starting with 0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129 not found: ID does not exist" containerID="0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129" Aug 04 12:41:02.482447 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.482367 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129"} err="failed to get container status \"0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129\": rpc error: code = NotFound desc = could not find container \"0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129\": container with ID starting with 0646c540969ee22d8553510e96a2487ffe491f47f04a36bed922a910f15b0129 not found: ID does not exist" Aug 04 12:41:02.496037 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.496010 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d6f5ff99-qg6g5"] Aug 04 12:41:02.501019 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:02.500999 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d6f5ff99-qg6g5"] Aug 04 12:41:03.176630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:03.176566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:41:03.553012 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:03.552934 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cab951c-a22f-4555-8557-4d01293bed43" path="/var/lib/kubelet/pods/9cab951c-a22f-4555-8557-4d01293bed43/volumes" Aug 04 12:41:03.847306 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:03.847273 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:41:49.528097 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.528062 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-drqk6"] Aug 04 12:41:49.528643 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.528333 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d8768cdd-drqk6" podUID="841dbb8a-f049-4e7a-ba52-968932a44c60" containerName="manager" containerID="cri-o://cc08cfcca35f69754d9741f1c618cb81fa5db1a7ae5225a83340067cd5074155" gracePeriod=30 Aug 04 12:41:49.698226 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.698183 2578 generic.go:356] "Generic (PLEG): container finished" podID="841dbb8a-f049-4e7a-ba52-968932a44c60" containerID="cc08cfcca35f69754d9741f1c618cb81fa5db1a7ae5225a83340067cd5074155" exitCode=0 Aug 04 12:41:49.698393 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.698248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-drqk6" event={"ID":"841dbb8a-f049-4e7a-ba52-968932a44c60","Type":"ContainerDied","Data":"cc08cfcca35f69754d9741f1c618cb81fa5db1a7ae5225a83340067cd5074155"} Aug 04 12:41:49.788604 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.788580 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:41:49.822518 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.822494 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert\") pod \"841dbb8a-f049-4e7a-ba52-968932a44c60\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " Aug 04 12:41:49.822689 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.822536 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sk9q\" (UniqueName: \"kubernetes.io/projected/841dbb8a-f049-4e7a-ba52-968932a44c60-kube-api-access-5sk9q\") pod \"841dbb8a-f049-4e7a-ba52-968932a44c60\" (UID: \"841dbb8a-f049-4e7a-ba52-968932a44c60\") " Aug 04 12:41:49.824898 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.824860 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "841dbb8a-f049-4e7a-ba52-968932a44c60" (UID: "841dbb8a-f049-4e7a-ba52-968932a44c60"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:41:49.825020 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.824955 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841dbb8a-f049-4e7a-ba52-968932a44c60-kube-api-access-5sk9q" (OuterVolumeSpecName: "kube-api-access-5sk9q") pod "841dbb8a-f049-4e7a-ba52-968932a44c60" (UID: "841dbb8a-f049-4e7a-ba52-968932a44c60"). InnerVolumeSpecName "kube-api-access-5sk9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:41:49.923982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.923946 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/841dbb8a-f049-4e7a-ba52-968932a44c60-webhook-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:41:49.923982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:49.923979 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sk9q\" (UniqueName: \"kubernetes.io/projected/841dbb8a-f049-4e7a-ba52-968932a44c60-kube-api-access-5sk9q\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:41:50.704474 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:50.704445 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-drqk6" Aug 04 12:41:50.704474 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:50.704465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-drqk6" event={"ID":"841dbb8a-f049-4e7a-ba52-968932a44c60","Type":"ContainerDied","Data":"9a3d569d69f821d4a9130225add5bc86ccc35c943cf91050348df63e111d2f10"} Aug 04 12:41:50.704986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:50.704519 2578 scope.go:117] "RemoveContainer" containerID="cc08cfcca35f69754d9741f1c618cb81fa5db1a7ae5225a83340067cd5074155" Aug 04 12:41:50.728317 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:50.728288 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-drqk6"] Aug 04 12:41:50.732498 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:50.732475 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-drqk6"] Aug 04 12:41:51.554469 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:41:51.554431 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841dbb8a-f049-4e7a-ba52-968932a44c60" path="/var/lib/kubelet/pods/841dbb8a-f049-4e7a-ba52-968932a44c60/volumes" Aug 04 12:42:11.344854 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:11.344820 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:42:11.445982 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:11.445948 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:42:49.393864 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:49.393830 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:42:49.453612 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:49.453586 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:42:49.454979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:49.454951 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:42:49.495494 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:49.495453 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:42:51.528692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.528654 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d8768cdd-66hbj"] Aug 04 12:42:51.529082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.529018 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841dbb8a-f049-4e7a-ba52-968932a44c60" containerName="manager" Aug 04 12:42:51.529082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.529029 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="841dbb8a-f049-4e7a-ba52-968932a44c60" containerName="manager" Aug 04 12:42:51.529082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.529045 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cab951c-a22f-4555-8557-4d01293bed43" containerName="authorino" Aug 04 12:42:51.529082 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.529050 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cab951c-a22f-4555-8557-4d01293bed43" containerName="authorino" Aug 04 12:42:51.529256 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.529150 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cab951c-a22f-4555-8557-4d01293bed43" containerName="authorino" Aug 04 12:42:51.529256 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.529163 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="841dbb8a-f049-4e7a-ba52-968932a44c60" containerName="manager" Aug 04 12:42:51.533374 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.533353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.536367 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.536338 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cw2b9\"" Aug 04 12:42:51.536484 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.536409 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 12:42:51.536484 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.536430 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 12:42:51.542192 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.542168 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-66hbj"] Aug 04 12:42:51.700055 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.700022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9adf70ea-8310-40e0-a740-b88c73678456-webhook-cert\") pod \"maas-controller-68d8768cdd-66hbj\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.700263 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.700183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwkx\" (UniqueName: \"kubernetes.io/projected/9adf70ea-8310-40e0-a740-b88c73678456-kube-api-access-bhwkx\") pod \"maas-controller-68d8768cdd-66hbj\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.800728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.800621 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwkx\" (UniqueName: \"kubernetes.io/projected/9adf70ea-8310-40e0-a740-b88c73678456-kube-api-access-bhwkx\") pod \"maas-controller-68d8768cdd-66hbj\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.800728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.800672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9adf70ea-8310-40e0-a740-b88c73678456-webhook-cert\") pod \"maas-controller-68d8768cdd-66hbj\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.803508 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.803484 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9adf70ea-8310-40e0-a740-b88c73678456-webhook-cert\") pod \"maas-controller-68d8768cdd-66hbj\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.811182 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.811153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwkx\" (UniqueName: \"kubernetes.io/projected/9adf70ea-8310-40e0-a740-b88c73678456-kube-api-access-bhwkx\") pod \"maas-controller-68d8768cdd-66hbj\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:51.844682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:51.844647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:52.190763 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:52.190735 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-66hbj"] Aug 04 12:42:52.191828 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:42:52.191802 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adf70ea_8310_40e0_a740_b88c73678456.slice/crio-017438fc2bc0540fce10b56d551ec6f2b22be3e2179f9ee5a3c2c5694ac92344 WatchSource:0}: Error finding container 017438fc2bc0540fce10b56d551ec6f2b22be3e2179f9ee5a3c2c5694ac92344: Status 404 returned error can't find the container with id 017438fc2bc0540fce10b56d551ec6f2b22be3e2179f9ee5a3c2c5694ac92344 Aug 04 12:42:53.004740 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:53.004698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-66hbj" event={"ID":"9adf70ea-8310-40e0-a740-b88c73678456","Type":"ContainerStarted","Data":"40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f"} Aug 04 12:42:53.005161 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:53.004756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-66hbj" event={"ID":"9adf70ea-8310-40e0-a740-b88c73678456","Type":"ContainerStarted","Data":"017438fc2bc0540fce10b56d551ec6f2b22be3e2179f9ee5a3c2c5694ac92344"} Aug 04 12:42:53.005161 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:53.004785 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:42:53.032533 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:53.032423 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d8768cdd-66hbj" podStartSLOduration=1.448936872 podStartE2EDuration="2.032400895s" totalImagesPullingTime="583.464023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:42:53.031152962 +0000 UTC m=+903.986748757" watchObservedRunningTime="2026-08-04 12:42:53.032400895 +0000 UTC m=+903.987996690" Aug 04 12:42:59.268988 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:42:59.268952 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:43:01.266512 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:01.266476 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:43:04.011793 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:04.011759 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:43:08.513230 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.513191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:43:08.610465 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.610430 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-66hbj"] Aug 04 12:43:08.610660 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.610626 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d8768cdd-66hbj" podUID="9adf70ea-8310-40e0-a740-b88c73678456" containerName="manager" containerID="cri-o://40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f" gracePeriod=30 Aug 04 12:43:08.855250 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.855224 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:43:08.858392 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.858372 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9adf70ea-8310-40e0-a740-b88c73678456-webhook-cert\") pod \"9adf70ea-8310-40e0-a740-b88c73678456\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " Aug 04 12:43:08.858489 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.858425 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhwkx\" (UniqueName: \"kubernetes.io/projected/9adf70ea-8310-40e0-a740-b88c73678456-kube-api-access-bhwkx\") pod \"9adf70ea-8310-40e0-a740-b88c73678456\" (UID: \"9adf70ea-8310-40e0-a740-b88c73678456\") " Aug 04 12:43:08.860581 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.860550 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adf70ea-8310-40e0-a740-b88c73678456-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9adf70ea-8310-40e0-a740-b88c73678456" (UID: "9adf70ea-8310-40e0-a740-b88c73678456"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:43:08.860703 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.860686 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adf70ea-8310-40e0-a740-b88c73678456-kube-api-access-bhwkx" (OuterVolumeSpecName: "kube-api-access-bhwkx") pod "9adf70ea-8310-40e0-a740-b88c73678456" (UID: "9adf70ea-8310-40e0-a740-b88c73678456"). InnerVolumeSpecName "kube-api-access-bhwkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:43:08.959641 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.959598 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9adf70ea-8310-40e0-a740-b88c73678456-webhook-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:43:08.959641 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:08.959632 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhwkx\" (UniqueName: \"kubernetes.io/projected/9adf70ea-8310-40e0-a740-b88c73678456-kube-api-access-bhwkx\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:43:09.081698 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.081608 2578 generic.go:356] "Generic (PLEG): container finished" podID="9adf70ea-8310-40e0-a740-b88c73678456" containerID="40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f" exitCode=0 Aug 04 12:43:09.081698 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.081658 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-66hbj" Aug 04 12:43:09.081871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.081701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-66hbj" event={"ID":"9adf70ea-8310-40e0-a740-b88c73678456","Type":"ContainerDied","Data":"40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f"} Aug 04 12:43:09.081871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.081777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-66hbj" event={"ID":"9adf70ea-8310-40e0-a740-b88c73678456","Type":"ContainerDied","Data":"017438fc2bc0540fce10b56d551ec6f2b22be3e2179f9ee5a3c2c5694ac92344"} Aug 04 12:43:09.081871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.081800 2578 scope.go:117] "RemoveContainer" containerID="40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f" Aug 04 12:43:09.091341 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.091321 2578 scope.go:117] "RemoveContainer" containerID="40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f" Aug 04 12:43:09.091607 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:43:09.091588 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f\": container with ID starting with 40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f not found: ID does not exist" containerID="40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f" Aug 04 12:43:09.091661 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.091615 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f"} err="failed to get container status \"40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f\": rpc error: code = NotFound desc = could not find container \"40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f\": container with ID starting with 40e85e9e3b1d6f8f1668c3c746f74aa1bb45680c379f3994d95df249e3fbee4f not found: ID does not exist" Aug 04 12:43:09.104383 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.104352 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-66hbj"] Aug 04 12:43:09.109333 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.109308 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-66hbj"] Aug 04 12:43:09.553992 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.553958 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adf70ea-8310-40e0-a740-b88c73678456" path="/var/lib/kubelet/pods/9adf70ea-8310-40e0-a740-b88c73678456/volumes" Aug 04 12:43:09.748750 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.748696 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:43:09.854467 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.854430 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d8768cdd-c7g2g"] Aug 04 12:43:09.854834 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.854821 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adf70ea-8310-40e0-a740-b88c73678456" containerName="manager" Aug 04 12:43:09.854878 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.854836 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adf70ea-8310-40e0-a740-b88c73678456" containerName="manager" Aug 04 12:43:09.854931 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.854921 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adf70ea-8310-40e0-a740-b88c73678456" containerName="manager" Aug 04 12:43:09.859526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.859510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:09.862460 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.862440 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 12:43:09.863651 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.863634 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cw2b9\"" Aug 04 12:43:09.863736 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.863656 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 12:43:09.867499 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.867482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6598625-e743-4b38-8ef5-7f62371e40e8-webhook-cert\") pod \"maas-controller-68d8768cdd-c7g2g\" (UID: \"b6598625-e743-4b38-8ef5-7f62371e40e8\") " pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:09.867593 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.867514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpsn\" (UniqueName: \"kubernetes.io/projected/b6598625-e743-4b38-8ef5-7f62371e40e8-kube-api-access-cwpsn\") pod \"maas-controller-68d8768cdd-c7g2g\" (UID: \"b6598625-e743-4b38-8ef5-7f62371e40e8\") " pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:09.869358 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.869334 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-c7g2g"] Aug 04 12:43:09.968015 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.967974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpsn\" (UniqueName: \"kubernetes.io/projected/b6598625-e743-4b38-8ef5-7f62371e40e8-kube-api-access-cwpsn\") pod \"maas-controller-68d8768cdd-c7g2g\" (UID: \"b6598625-e743-4b38-8ef5-7f62371e40e8\") " pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:09.968237 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.968107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6598625-e743-4b38-8ef5-7f62371e40e8-webhook-cert\") pod \"maas-controller-68d8768cdd-c7g2g\" (UID: \"b6598625-e743-4b38-8ef5-7f62371e40e8\") " pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:09.970634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.970616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6598625-e743-4b38-8ef5-7f62371e40e8-webhook-cert\") pod \"maas-controller-68d8768cdd-c7g2g\" (UID: \"b6598625-e743-4b38-8ef5-7f62371e40e8\") " pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:09.978269 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:09.978250 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpsn\" (UniqueName: \"kubernetes.io/projected/b6598625-e743-4b38-8ef5-7f62371e40e8-kube-api-access-cwpsn\") pod \"maas-controller-68d8768cdd-c7g2g\" (UID: \"b6598625-e743-4b38-8ef5-7f62371e40e8\") " pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:10.170970 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:10.170879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:10.307571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:10.307546 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d8768cdd-c7g2g"] Aug 04 12:43:10.309896 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:43:10.309869 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6598625_e743_4b38_8ef5_7f62371e40e8.slice/crio-f79c173452aced6cdc8fdc8858816641f6495b1a88e825260608ee3cba0e7db4 WatchSource:0}: Error finding container f79c173452aced6cdc8fdc8858816641f6495b1a88e825260608ee3cba0e7db4: Status 404 returned error can't find the container with id f79c173452aced6cdc8fdc8858816641f6495b1a88e825260608ee3cba0e7db4 Aug 04 12:43:11.093831 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:11.093798 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" event={"ID":"b6598625-e743-4b38-8ef5-7f62371e40e8","Type":"ContainerStarted","Data":"b0d02489d777935c0a1f72293b297e09546181e11af2117182faad1d3c4e6218"} Aug 04 12:43:11.094255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:11.093854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" event={"ID":"b6598625-e743-4b38-8ef5-7f62371e40e8","Type":"ContainerStarted","Data":"f79c173452aced6cdc8fdc8858816641f6495b1a88e825260608ee3cba0e7db4"} Aug 04 12:43:11.094255 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:11.093899 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:43:11.116245 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:11.116196 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" podStartSLOduration=1.784826009 podStartE2EDuration="2.11618055s" totalImagesPullingTime="331.354541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:43:11.113550437 +0000 UTC m=+922.069146246" watchObservedRunningTime="2026-08-04 12:43:11.11618055 +0000 UTC m=+922.071776371" Aug 04 12:43:22.101913 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:43:22.101884 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d8768cdd-c7g2g" Aug 04 12:44:08.114292 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:08.114259 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:08.207439 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:08.207406 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:08.330104 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:08.330071 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:14.570029 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:14.569947 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:14.671027 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:14.670994 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:14.771172 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:14.771112 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:14.907096 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:14.907056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:21.151437 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:21.151401 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:44:21.216463 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:44:21.216433 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:45:00.146904 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.146866 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t"] Aug 04 12:45:00.150495 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.150477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" Aug 04 12:45:00.153372 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.153349 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sbvbg\"" Aug 04 12:45:00.160053 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.160034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" Aug 04 12:45:00.168452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.168428 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t"] Aug 04 12:45:00.302353 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.302319 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t"] Aug 04 12:45:00.304309 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:45:00.304281 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442abb3b_6308_473c_8864_2f86d1e65670.slice/crio-e21fcf4c082bd7733133ed6b994da5d43cac62ec32428568f4b2251a5a44659f WatchSource:0}: Error finding container e21fcf4c082bd7733133ed6b994da5d43cac62ec32428568f4b2251a5a44659f: Status 404 returned error can't find the container with id e21fcf4c082bd7733133ed6b994da5d43cac62ec32428568f4b2251a5a44659f Aug 04 12:45:00.306394 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.306375 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 12:45:00.641043 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:00.641010 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" event={"ID":"442abb3b-6308-473c-8864-2f86d1e65670","Type":"ContainerStarted","Data":"e21fcf4c082bd7733133ed6b994da5d43cac62ec32428568f4b2251a5a44659f"} Aug 04 12:45:03.656786 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:03.656747 2578 generic.go:356] "Generic (PLEG): container finished" podID="442abb3b-6308-473c-8864-2f86d1e65670" containerID="c91bcda711ee42cfdec45cfbd0ef2fa7e3fd92d09e0e810c76351c9e3b1daf96" exitCode=0 Aug 04 12:45:03.657190 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:03.656818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" event={"ID":"442abb3b-6308-473c-8864-2f86d1e65670","Type":"ContainerDied","Data":"c91bcda711ee42cfdec45cfbd0ef2fa7e3fd92d09e0e810c76351c9e3b1daf96"} Aug 04 12:45:04.790036 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:04.790012 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" Aug 04 12:45:05.669760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:05.669722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" event={"ID":"442abb3b-6308-473c-8864-2f86d1e65670","Type":"ContainerDied","Data":"e21fcf4c082bd7733133ed6b994da5d43cac62ec32428568f4b2251a5a44659f"} Aug 04 12:45:05.669760 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:05.669756 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764125-grz8t" Aug 04 12:45:05.669984 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:05.669780 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e21fcf4c082bd7733133ed6b994da5d43cac62ec32428568f4b2251a5a44659f" Aug 04 12:45:56.934077 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:45:56.933995 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:47:49.486206 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:47:49.486178 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:47:49.487660 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:47:49.487638 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:50:09.973987 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:50:09.973906 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:51:01.227304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:51:01.227273 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:52:02.331840 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:52:02.331753 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:52:49.517525 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:52:49.517494 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:52:49.520414 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:52:49.520390 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:53:13.275788 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:13.275709 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:53:49.107956 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.107915 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb"] Aug 04 12:53:49.108462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.108329 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442abb3b-6308-473c-8864-2f86d1e65670" containerName="cleanup" Aug 04 12:53:49.108462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.108341 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="442abb3b-6308-473c-8864-2f86d1e65670" containerName="cleanup" Aug 04 12:53:49.108462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.108408 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="442abb3b-6308-473c-8864-2f86d1e65670" containerName="cleanup" Aug 04 12:53:49.111552 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.111533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.114518 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.114497 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-1940871b-kserve-self-signed-certs\"" Aug 04 12:53:49.133245 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.133217 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb"] Aug 04 12:53:49.144361 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.144323 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:53:49.197580 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.197542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-dshm\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.197580 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.197581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tls-certs\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.197798 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.197618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tmp-dir\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.197798 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.197663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-home\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.197798 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.197758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-model-cache\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.197901 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.197817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtjx\" (UniqueName: \"kubernetes.io/projected/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-kube-api-access-mbtjx\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298281 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-home\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298281 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-model-cache\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298540 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtjx\" (UniqueName: \"kubernetes.io/projected/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-kube-api-access-mbtjx\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298540 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-dshm\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298540 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tls-certs\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298540 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tmp-dir\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298752 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-home\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298818 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-model-cache\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.298893 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.298872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tmp-dir\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.300758 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.300737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-dshm\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.300985 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.300969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tls-certs\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.308465 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.308443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtjx\" (UniqueName: \"kubernetes.io/projected/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-kube-api-access-mbtjx\") pod \"e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.421440 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.421354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:49.558802 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:49.558781 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb"] Aug 04 12:53:49.560794 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:53:49.560764 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf71bdb97_2e7a_43a6_afa8_98fd2e3be5a8.slice/crio-699997ceb07906b199bcec19ec2249bfe717e41c1fdccd008ea5ca9b9d6dd967 WatchSource:0}: Error finding container 699997ceb07906b199bcec19ec2249bfe717e41c1fdccd008ea5ca9b9d6dd967: Status 404 returned error can't find the container with id 699997ceb07906b199bcec19ec2249bfe717e41c1fdccd008ea5ca9b9d6dd967 Aug 04 12:53:50.252402 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:50.252357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" event={"ID":"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8","Type":"ContainerStarted","Data":"1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f"} Aug 04 12:53:50.252402 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:50.252405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" event={"ID":"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8","Type":"ContainerStarted","Data":"699997ceb07906b199bcec19ec2249bfe717e41c1fdccd008ea5ca9b9d6dd967"} Aug 04 12:53:50.275001 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:50.274914 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" podStartSLOduration=1.2748975169999999 podStartE2EDuration="1.274897517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:53:50.27365803 +0000 UTC m=+1561.229253825" watchObservedRunningTime="2026-08-04 12:53:50.274897517 +0000 UTC m=+1561.230493311" Aug 04 12:53:57.214495 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:57.214457 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:53:59.421919 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:59.421827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:59.421919 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:59.421874 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:53:59.434427 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:53:59.434407 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:54:00.313102 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:00.313068 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:54:26.406871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:26.406835 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:54:40.171692 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:40.171607 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb"] Aug 04 12:54:40.203910 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:40.203877 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:54:41.398703 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:41.398670 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:54:48.949324 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:48.949286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:54:55.192939 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.192897 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" podUID="f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" containerName="main" containerID="cri-o://1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f" gracePeriod=45 Aug 04 12:54:55.433825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.433797 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:54:55.550179 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550071 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tmp-dir\") pod \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " Aug 04 12:54:55.550179 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550114 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-model-cache\") pod \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " Aug 04 12:54:55.550421 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550187 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-home\") pod \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " Aug 04 12:54:55.550421 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550206 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbtjx\" (UniqueName: \"kubernetes.io/projected/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-kube-api-access-mbtjx\") pod \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " Aug 04 12:54:55.550421 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550263 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tls-certs\") pod \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " Aug 04 12:54:55.550421 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550342 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-dshm\") pod \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\" (UID: \"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8\") " Aug 04 12:54:55.550421 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550355 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" (UID: "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:54:55.550701 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550468 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-home" (OuterVolumeSpecName: "home") pod "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" (UID: "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:54:55.550701 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550498 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" (UID: "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:54:55.550701 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550700 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tmp-dir\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:54:55.550816 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550713 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-model-cache\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:54:55.550816 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.550725 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-home\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:54:55.553038 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.553011 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-dshm" (OuterVolumeSpecName: "dshm") pod "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" (UID: "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 12:54:55.553212 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.553147 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-kube-api-access-mbtjx" (OuterVolumeSpecName: "kube-api-access-mbtjx") pod "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" (UID: "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8"). InnerVolumeSpecName "kube-api-access-mbtjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:54:55.553212 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.553157 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" (UID: "f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:54:55.581825 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.581798 2578 generic.go:356] "Generic (PLEG): container finished" podID="f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" containerID="1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f" exitCode=0 Aug 04 12:54:55.581986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.581845 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" Aug 04 12:54:55.581986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.581889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" event={"ID":"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8","Type":"ContainerDied","Data":"1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f"} Aug 04 12:54:55.581986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.581933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb" event={"ID":"f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8","Type":"ContainerDied","Data":"699997ceb07906b199bcec19ec2249bfe717e41c1fdccd008ea5ca9b9d6dd967"} Aug 04 12:54:55.581986 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.581950 2578 scope.go:117] "RemoveContainer" containerID="1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f" Aug 04 12:54:55.590827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.590810 2578 scope.go:117] "RemoveContainer" containerID="1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f" Aug 04 12:54:55.591144 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:54:55.591099 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f\": container with ID starting with 1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f not found: ID does not exist" containerID="1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f" Aug 04 12:54:55.591205 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.591158 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f"} err="failed to get container status \"1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f\": rpc error: code = NotFound desc = could not find container \"1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f\": container with ID starting with 1d70fdb977953ffd6d802eefc693383d43e65ef9ec6ec0e1f8632b52a7740e2f not found: ID does not exist" Aug 04 12:54:55.602577 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.602546 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb"] Aug 04 12:54:55.607141 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.607101 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-1940871b-kserve-5f78f78c4c-rw6qb"] Aug 04 12:54:55.651594 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.651560 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbtjx\" (UniqueName: \"kubernetes.io/projected/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-kube-api-access-mbtjx\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:54:55.651594 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.651590 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-tls-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:54:55.651594 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:55.651601 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8-dshm\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:54:57.553571 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:54:57.553538 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" path="/var/lib/kubelet/pods/f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8/volumes" Aug 04 12:57:49.549452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:57:49.549423 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:57:49.555329 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:57:49.555302 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 12:58:04.059588 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:04.059539 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:30.078030 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:30.077957 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:33.247572 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:33.247526 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:33.346321 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:33.346290 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:33.727240 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:33.727203 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:39.901649 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:39.901616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:39.973600 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:39.973538 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:40.141161 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:40.141113 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:40.462109 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:40.462053 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:40.911559 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:40.911527 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:42.963401 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:42.963361 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:43.061505 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:43.061450 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:44.171185 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.171145 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf"] Aug 04 12:58:44.171741 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.171712 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" containerName="main" Aug 04 12:58:44.171741 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.171738 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" containerName="main" Aug 04 12:58:44.171890 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.171821 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71bdb97-2e7a-43a6-afa8-98fd2e3be5a8" containerName="main" Aug 04 12:58:44.175026 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.175006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.177841 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.177820 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a1756385-serving-cert\"" Aug 04 12:58:44.186357 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.186292 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf"] Aug 04 12:58:44.211936 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.211908 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:44.250358 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.250324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.250526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.250363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b636de9f-f2b9-4c85-8e76-550d3372e515-openshift-service-ca\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.250526 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.250398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kjs\" (UniqueName: \"kubernetes.io/projected/b636de9f-f2b9-4c85-8e76-550d3372e515-kube-api-access-t4kjs\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.351660 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.351625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kjs\" (UniqueName: \"kubernetes.io/projected/b636de9f-f2b9-4c85-8e76-550d3372e515-kube-api-access-t4kjs\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.351822 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.351734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.351822 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.351763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b636de9f-f2b9-4c85-8e76-550d3372e515-openshift-service-ca\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.351892 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:58:44.351882 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-serving-cert: secret "maas-api-e2e-ait-a1756385-serving-cert" not found Aug 04 12:58:44.351960 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:58:44.351948 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls podName:b636de9f-f2b9-4c85-8e76-550d3372e515 nodeName:}" failed. No retries permitted until 2026-08-04 12:58:44.851929356 +0000 UTC m=+1855.807525127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls") pod "maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" (UID: "b636de9f-f2b9-4c85-8e76-550d3372e515") : secret "maas-api-e2e-ait-a1756385-serving-cert" not found Aug 04 12:58:44.352484 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.352464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b636de9f-f2b9-4c85-8e76-550d3372e515-openshift-service-ca\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.362160 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.362089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kjs\" (UniqueName: \"kubernetes.io/projected/b636de9f-f2b9-4c85-8e76-550d3372e515-kube-api-access-t4kjs\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.434508 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.434396 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:44.718871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.718782 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:44.820913 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.820877 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:58:44.856767 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.856737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.859318 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.859297 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls\") pod \"maas-api-e2e-ait-a1756385-745d757cd6-pd2kf\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:44.866053 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.866029 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l"] Aug 04 12:58:44.870008 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.869992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:44.872932 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.872911 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a1756385-dockercfg-47sbl\"" Aug 04 12:58:44.873251 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.873231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a1756385\"" Aug 04 12:58:44.880630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.880608 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l"] Aug 04 12:58:44.903978 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.903943 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh"] Aug 04 12:58:44.907521 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.907503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:44.918570 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.918549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh"] Aug 04 12:58:44.957952 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.957923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c952bdc-d938-4137-8378-f4a142540138-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:44.958214 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:44.958031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qrm\" (UniqueName: \"kubernetes.io/projected/4c952bdc-d938-4137-8378-f4a142540138-kube-api-access-f9qrm\") pod \"payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.059182 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.059074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qrm\" (UniqueName: \"kubernetes.io/projected/4c952bdc-d938-4137-8378-f4a142540138-kube-api-access-f9qrm\") pod \"payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.059372 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.059196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d635df-ab28-44fd-9a7c-9bf279ae244c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.059372 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.059223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqx9\" (UniqueName: \"kubernetes.io/projected/44d635df-ab28-44fd-9a7c-9bf279ae244c-kube-api-access-fpqx9\") pod \"payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.059372 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.059266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c952bdc-d938-4137-8378-f4a142540138-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.059978 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.059959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c952bdc-d938-4137-8378-f4a142540138-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.070501 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.070479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qrm\" (UniqueName: \"kubernetes.io/projected/4c952bdc-d938-4137-8378-f4a142540138-kube-api-access-f9qrm\") pod \"payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.087392 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.087368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:45.160702 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.160674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d635df-ab28-44fd-9a7c-9bf279ae244c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.160855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.160708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqx9\" (UniqueName: \"kubernetes.io/projected/44d635df-ab28-44fd-9a7c-9bf279ae244c-kube-api-access-fpqx9\") pod \"payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.161532 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.161498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d635df-ab28-44fd-9a7c-9bf279ae244c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.175789 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.175758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqx9\" (UniqueName: \"kubernetes.io/projected/44d635df-ab28-44fd-9a7c-9bf279ae244c-kube-api-access-fpqx9\") pod \"payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.181335 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.181309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.218634 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.218602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.232812 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.232732 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf"] Aug 04 12:58:45.235990 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:58:45.235963 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb636de9f_f2b9_4c85_8e76_550d3372e515.slice/crio-659d9826bd4d3f44a163f7bb6e21dd847c995614b237ef23bacefb01b7c9c891 WatchSource:0}: Error finding container 659d9826bd4d3f44a163f7bb6e21dd847c995614b237ef23bacefb01b7c9c891: Status 404 returned error can't find the container with id 659d9826bd4d3f44a163f7bb6e21dd847c995614b237ef23bacefb01b7c9c891 Aug 04 12:58:45.237890 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.237866 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 12:58:45.344607 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.344565 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l"] Aug 04 12:58:45.346365 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:58:45.346336 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c952bdc_d938_4137_8378_f4a142540138.slice/crio-ae04fccc6ad09461715defb2ffc1b56e0ee1724ae151f8e1c4f226cc96fb24a2 WatchSource:0}: Error finding container ae04fccc6ad09461715defb2ffc1b56e0ee1724ae151f8e1c4f226cc96fb24a2: Status 404 returned error can't find the container with id ae04fccc6ad09461715defb2ffc1b56e0ee1724ae151f8e1c4f226cc96fb24a2 Aug 04 12:58:45.397425 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.397393 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh"] Aug 04 12:58:45.399322 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:58:45.399301 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d635df_ab28_44fd_9a7c_9bf279ae244c.slice/crio-0ae88d73e48063594031500dea81f75cfa2635db2d68a7bcf2b3c125c8c184e1 WatchSource:0}: Error finding container 0ae88d73e48063594031500dea81f75cfa2635db2d68a7bcf2b3c125c8c184e1: Status 404 returned error can't find the container with id 0ae88d73e48063594031500dea81f75cfa2635db2d68a7bcf2b3c125c8c184e1 Aug 04 12:58:45.722781 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.722738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" event={"ID":"44d635df-ab28-44fd-9a7c-9bf279ae244c","Type":"ContainerStarted","Data":"bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060"} Aug 04 12:58:45.723001 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.722832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" event={"ID":"44d635df-ab28-44fd-9a7c-9bf279ae244c","Type":"ContainerStarted","Data":"0ae88d73e48063594031500dea81f75cfa2635db2d68a7bcf2b3c125c8c184e1"} Aug 04 12:58:45.723001 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.722864 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:45.724442 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.724415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" event={"ID":"4c952bdc-d938-4137-8378-f4a142540138","Type":"ContainerStarted","Data":"985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3"} Aug 04 12:58:45.724545 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.724468 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" event={"ID":"4c952bdc-d938-4137-8378-f4a142540138","Type":"ContainerStarted","Data":"ae04fccc6ad09461715defb2ffc1b56e0ee1724ae151f8e1c4f226cc96fb24a2"} Aug 04 12:58:45.724545 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.724496 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:58:45.725561 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.725532 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" event={"ID":"b636de9f-f2b9-4c85-8e76-550d3372e515","Type":"ContainerStarted","Data":"659d9826bd4d3f44a163f7bb6e21dd847c995614b237ef23bacefb01b7c9c891"} Aug 04 12:58:45.770617 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.770568 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" podStartSLOduration=1.770551592 podStartE2EDuration="1.770551592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:58:45.770169632 +0000 UTC m=+1856.725765427" watchObservedRunningTime="2026-08-04 12:58:45.770551592 +0000 UTC m=+1856.726147390" Aug 04 12:58:45.772016 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:45.771941 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" podStartSLOduration=1.771932788 podStartE2EDuration="1.771932788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:58:45.743563216 +0000 UTC m=+1856.699159007" watchObservedRunningTime="2026-08-04 12:58:45.771932788 +0000 UTC m=+1856.727528581" Aug 04 12:58:47.737709 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:47.737679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" event={"ID":"b636de9f-f2b9-4c85-8e76-550d3372e515","Type":"ContainerStarted","Data":"3cf1cdf4906388386837c22b9ef2e88250e7cd75edb51ad1bb4d00d1674fd4f3"} Aug 04 12:58:47.738219 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:47.737825 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:47.761233 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:47.761172 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" podStartSLOduration=2.208074764 podStartE2EDuration="3.761151443s" totalImagesPullingTime="1.553076679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:58:47.76084595 +0000 UTC m=+1858.716441745" watchObservedRunningTime="2026-08-04 12:58:47.761151443 +0000 UTC m=+1858.716747237" Aug 04 12:58:53.748728 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:53.748699 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:58:56.733405 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:56.733377 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:58:56.733779 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:58:56.733445 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:59:24.755559 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.755484 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf"] Aug 04 12:59:24.756042 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.755684 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" podUID="b636de9f-f2b9-4c85-8e76-550d3372e515" containerName="maas-api" containerID="cri-o://3cf1cdf4906388386837c22b9ef2e88250e7cd75edb51ad1bb4d00d1674fd4f3" gracePeriod=30 Aug 04 12:59:24.850398 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.850369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:24.854363 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.854337 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l"] Aug 04 12:59:24.854630 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.854588 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" podUID="4c952bdc-d938-4137-8378-f4a142540138" containerName="payload-processing" containerID="cri-o://985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3" gracePeriod=30 Aug 04 12:59:24.866264 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.866236 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh"] Aug 04 12:59:24.866667 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.866620 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" podUID="44d635df-ab28-44fd-9a7c-9bf279ae244c" containerName="payload-pre-processing" containerID="cri-o://bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060" gracePeriod=30 Aug 04 12:59:24.912748 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.912716 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:24.932827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.932765 2578 generic.go:356] "Generic (PLEG): container finished" podID="b636de9f-f2b9-4c85-8e76-550d3372e515" containerID="3cf1cdf4906388386837c22b9ef2e88250e7cd75edb51ad1bb4d00d1674fd4f3" exitCode=0 Aug 04 12:59:24.932827 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:24.932821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" event={"ID":"b636de9f-f2b9-4c85-8e76-550d3372e515","Type":"ContainerDied","Data":"3cf1cdf4906388386837c22b9ef2e88250e7cd75edb51ad1bb4d00d1674fd4f3"} Aug 04 12:59:25.018792 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.018664 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:25.022081 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.022058 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:59:25.115590 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.115562 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:59:25.141347 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.141322 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:59:25.153010 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.152980 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls\") pod \"b636de9f-f2b9-4c85-8e76-550d3372e515\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " Aug 04 12:59:25.153208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.153091 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b636de9f-f2b9-4c85-8e76-550d3372e515-openshift-service-ca\") pod \"b636de9f-f2b9-4c85-8e76-550d3372e515\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " Aug 04 12:59:25.153208 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.153142 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4kjs\" (UniqueName: \"kubernetes.io/projected/b636de9f-f2b9-4c85-8e76-550d3372e515-kube-api-access-t4kjs\") pod \"b636de9f-f2b9-4c85-8e76-550d3372e515\" (UID: \"b636de9f-f2b9-4c85-8e76-550d3372e515\") " Aug 04 12:59:25.153535 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.153492 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b636de9f-f2b9-4c85-8e76-550d3372e515-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b636de9f-f2b9-4c85-8e76-550d3372e515" (UID: "b636de9f-f2b9-4c85-8e76-550d3372e515"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:59:25.157989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.157930 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b636de9f-f2b9-4c85-8e76-550d3372e515" (UID: "b636de9f-f2b9-4c85-8e76-550d3372e515"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 12:59:25.157989 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.157965 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b636de9f-f2b9-4c85-8e76-550d3372e515-kube-api-access-t4kjs" (OuterVolumeSpecName: "kube-api-access-t4kjs") pod "b636de9f-f2b9-4c85-8e76-550d3372e515" (UID: "b636de9f-f2b9-4c85-8e76-550d3372e515"). InnerVolumeSpecName "kube-api-access-t4kjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:59:25.199702 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.199672 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:25.254284 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254251 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9qrm\" (UniqueName: \"kubernetes.io/projected/4c952bdc-d938-4137-8378-f4a142540138-kube-api-access-f9qrm\") pod \"4c952bdc-d938-4137-8378-f4a142540138\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " Aug 04 12:59:25.254452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254291 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpqx9\" (UniqueName: \"kubernetes.io/projected/44d635df-ab28-44fd-9a7c-9bf279ae244c-kube-api-access-fpqx9\") pod \"44d635df-ab28-44fd-9a7c-9bf279ae244c\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " Aug 04 12:59:25.254452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254325 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c952bdc-d938-4137-8378-f4a142540138-plugins-config-volume\") pod \"4c952bdc-d938-4137-8378-f4a142540138\" (UID: \"4c952bdc-d938-4137-8378-f4a142540138\") " Aug 04 12:59:25.254452 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d635df-ab28-44fd-9a7c-9bf279ae244c-plugins-config-volume\") pod \"44d635df-ab28-44fd-9a7c-9bf279ae244c\" (UID: \"44d635df-ab28-44fd-9a7c-9bf279ae244c\") " Aug 04 12:59:25.254749 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254724 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b636de9f-f2b9-4c85-8e76-550d3372e515-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.254749 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254750 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b636de9f-f2b9-4c85-8e76-550d3372e515-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.254901 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254748 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c952bdc-d938-4137-8378-f4a142540138-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c952bdc-d938-4137-8378-f4a142540138" (UID: "4c952bdc-d938-4137-8378-f4a142540138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:59:25.254901 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254760 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4kjs\" (UniqueName: \"kubernetes.io/projected/b636de9f-f2b9-4c85-8e76-550d3372e515-kube-api-access-t4kjs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.254901 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.254760 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d635df-ab28-44fd-9a7c-9bf279ae244c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44d635df-ab28-44fd-9a7c-9bf279ae244c" (UID: "44d635df-ab28-44fd-9a7c-9bf279ae244c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 12:59:25.256701 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.256679 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d635df-ab28-44fd-9a7c-9bf279ae244c-kube-api-access-fpqx9" (OuterVolumeSpecName: "kube-api-access-fpqx9") pod "44d635df-ab28-44fd-9a7c-9bf279ae244c" (UID: "44d635df-ab28-44fd-9a7c-9bf279ae244c"). InnerVolumeSpecName "kube-api-access-fpqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:59:25.256768 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.256696 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c952bdc-d938-4137-8378-f4a142540138-kube-api-access-f9qrm" (OuterVolumeSpecName: "kube-api-access-f9qrm") pod "4c952bdc-d938-4137-8378-f4a142540138" (UID: "4c952bdc-d938-4137-8378-f4a142540138"). InnerVolumeSpecName "kube-api-access-f9qrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 12:59:25.355462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.355427 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qrm\" (UniqueName: \"kubernetes.io/projected/4c952bdc-d938-4137-8378-f4a142540138-kube-api-access-f9qrm\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.355462 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.355459 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqx9\" (UniqueName: \"kubernetes.io/projected/44d635df-ab28-44fd-9a7c-9bf279ae244c-kube-api-access-fpqx9\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.355654 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.355469 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c952bdc-d938-4137-8378-f4a142540138-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.355654 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.355483 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d635df-ab28-44fd-9a7c-9bf279ae244c-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 12:59:25.421863 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.421832 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:25.672935 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.672848 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:25.774899 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.774872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:25.903436 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.903410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:25.940667 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.940579 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" Aug 04 12:59:25.940667 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.940605 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf" event={"ID":"b636de9f-f2b9-4c85-8e76-550d3372e515","Type":"ContainerDied","Data":"659d9826bd4d3f44a163f7bb6e21dd847c995614b237ef23bacefb01b7c9c891"} Aug 04 12:59:25.940907 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.940670 2578 scope.go:117] "RemoveContainer" containerID="3cf1cdf4906388386837c22b9ef2e88250e7cd75edb51ad1bb4d00d1674fd4f3" Aug 04 12:59:25.942090 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.942067 2578 generic.go:356] "Generic (PLEG): container finished" podID="44d635df-ab28-44fd-9a7c-9bf279ae244c" containerID="bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060" exitCode=0 Aug 04 12:59:25.942225 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.942135 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" Aug 04 12:59:25.942356 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.942231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" event={"ID":"44d635df-ab28-44fd-9a7c-9bf279ae244c","Type":"ContainerDied","Data":"bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060"} Aug 04 12:59:25.942425 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.942387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh" event={"ID":"44d635df-ab28-44fd-9a7c-9bf279ae244c","Type":"ContainerDied","Data":"0ae88d73e48063594031500dea81f75cfa2635db2d68a7bcf2b3c125c8c184e1"} Aug 04 12:59:25.943778 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.943752 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c952bdc-d938-4137-8378-f4a142540138" containerID="985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3" exitCode=0 Aug 04 12:59:25.943875 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.943794 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" Aug 04 12:59:25.943875 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.943859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" event={"ID":"4c952bdc-d938-4137-8378-f4a142540138","Type":"ContainerDied","Data":"985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3"} Aug 04 12:59:25.943973 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.943899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l" event={"ID":"4c952bdc-d938-4137-8378-f4a142540138","Type":"ContainerDied","Data":"ae04fccc6ad09461715defb2ffc1b56e0ee1724ae151f8e1c4f226cc96fb24a2"} Aug 04 12:59:25.951269 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.951249 2578 scope.go:117] "RemoveContainer" containerID="bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060" Aug 04 12:59:25.959502 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.959480 2578 scope.go:117] "RemoveContainer" containerID="bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060" Aug 04 12:59:25.959776 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:59:25.959754 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060\": container with ID starting with bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060 not found: ID does not exist" containerID="bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060" Aug 04 12:59:25.959855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.959790 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060"} err="failed to get container status \"bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060\": rpc error: code = NotFound desc = could not find container \"bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060\": container with ID starting with bbeb3ef3b5d37b02811781a20e8c62915b5bc338f4893872152d1eeac02c6060 not found: ID does not exist" Aug 04 12:59:25.959855 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.959813 2578 scope.go:117] "RemoveContainer" containerID="985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3" Aug 04 12:59:25.968162 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.968032 2578 scope.go:117] "RemoveContainer" containerID="985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3" Aug 04 12:59:25.968467 ip-10-0-136-46 kubenswrapper[2578]: E0804 12:59:25.968439 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3\": container with ID starting with 985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3 not found: ID does not exist" containerID="985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3" Aug 04 12:59:25.968578 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.968476 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3"} err="failed to get container status \"985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3\": rpc error: code = NotFound desc = could not find container \"985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3\": container with ID starting with 985a1f44454c79a41d8f7fb0b80fc88c3b2ff67a37809df900831bcd670c97a3 not found: ID does not exist" Aug 04 12:59:25.970534 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.970513 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf"] Aug 04 12:59:25.980450 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.980426 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1756385-745d757cd6-pd2kf"] Aug 04 12:59:25.993002 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.992972 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l"] Aug 04 12:59:25.998183 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:25.998161 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1756385-67ffcb959b-bnx8l"] Aug 04 12:59:26.010167 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.010140 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh"] Aug 04 12:59:26.015102 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.015081 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1756385-84f4b6ddd-qn7dh"] Aug 04 12:59:26.182207 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182170 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r"] Aug 04 12:59:26.182621 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182603 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d635df-ab28-44fd-9a7c-9bf279ae244c" containerName="payload-pre-processing" Aug 04 12:59:26.182717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182624 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d635df-ab28-44fd-9a7c-9bf279ae244c" containerName="payload-pre-processing" Aug 04 12:59:26.182717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182636 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c952bdc-d938-4137-8378-f4a142540138" containerName="payload-processing" Aug 04 12:59:26.182717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182646 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c952bdc-d938-4137-8378-f4a142540138" containerName="payload-processing" Aug 04 12:59:26.182717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182675 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b636de9f-f2b9-4c85-8e76-550d3372e515" containerName="maas-api" Aug 04 12:59:26.182717 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182684 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b636de9f-f2b9-4c85-8e76-550d3372e515" containerName="maas-api" Aug 04 12:59:26.182960 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182794 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b636de9f-f2b9-4c85-8e76-550d3372e515" containerName="maas-api" Aug 04 12:59:26.182960 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182810 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d635df-ab28-44fd-9a7c-9bf279ae244c" containerName="payload-pre-processing" Aug 04 12:59:26.182960 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.182819 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c952bdc-d938-4137-8378-f4a142540138" containerName="payload-processing" Aug 04 12:59:26.187543 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.187521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.190923 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.190873 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 12:59:26.191007 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.190873 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fbddc9ad-gw-openshift-default-dockercfg-z2tmm\"" Aug 04 12:59:26.201171 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.201115 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r"] Aug 04 12:59:26.213914 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.213887 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:26.262597 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-envoy\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm28t\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-kube-api-access-tm28t\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-socket\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istiod-ca-cert\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-data\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262804 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-podinfo\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262871 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-certs\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-token\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.262980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.262930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-credential-socket\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-data\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-podinfo\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-certs\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-token\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-credential-socket\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-envoy\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm28t\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-kube-api-access-tm28t\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-data\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-socket\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istiod-ca-cert\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.364916 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-socket\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.365399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istiod-ca-cert\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.365926 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.365778 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-certs\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.366710 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.366357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-credential-socket\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.371818 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.368414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-podinfo\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.371818 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.369314 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-envoy\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.378327 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.378305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-token\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.378530 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.378510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm28t\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-kube-api-access-tm28t\") pod \"e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.398829 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.398801 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:26.498820 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.498731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:26.641813 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.641788 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r"] Aug 04 12:59:26.643829 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:59:26.643798 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742809a6_9e9f_4cd5_8c90_905ce4c0e5ce.slice/crio-a148e5784903b3b4c45e1f58bb64d890e0dc6ab9dffcc503c03982febaa27dfa WatchSource:0}: Error finding container a148e5784903b3b4c45e1f58bb64d890e0dc6ab9dffcc503c03982febaa27dfa: Status 404 returned error can't find the container with id a148e5784903b3b4c45e1f58bb64d890e0dc6ab9dffcc503c03982febaa27dfa Aug 04 12:59:26.950492 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:26.950451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" event={"ID":"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce","Type":"ContainerStarted","Data":"a148e5784903b3b4c45e1f58bb64d890e0dc6ab9dffcc503c03982febaa27dfa"} Aug 04 12:59:27.553582 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:27.553547 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d635df-ab28-44fd-9a7c-9bf279ae244c" path="/var/lib/kubelet/pods/44d635df-ab28-44fd-9a7c-9bf279ae244c/volumes" Aug 04 12:59:27.553925 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:27.553910 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c952bdc-d938-4137-8378-f4a142540138" path="/var/lib/kubelet/pods/4c952bdc-d938-4137-8378-f4a142540138/volumes" Aug 04 12:59:27.554288 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:27.554275 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b636de9f-f2b9-4c85-8e76-550d3372e515" path="/var/lib/kubelet/pods/b636de9f-f2b9-4c85-8e76-550d3372e515/volumes" Aug 04 12:59:29.117681 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:29.117637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:29.180083 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:29.180043 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:30.315413 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:30.315375 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 12:59:30.315821 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:30.315477 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 12:59:30.315821 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:30.315525 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 12:59:30.975629 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:30.975574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" event={"ID":"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce","Type":"ContainerStarted","Data":"631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce"} Aug 04 12:59:31.001320 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:31.001266 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podStartSLOduration=1.332125468 podStartE2EDuration="5.001250977s" totalImagesPullingTime="3.669125509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:59:30.99906064 +0000 UTC m=+1901.954656459" watchObservedRunningTime="2026-08-04 12:59:31.001250977 +0000 UTC m=+1901.956846771" Aug 04 12:59:31.498974 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:31.498930 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:31.504328 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:31.504299 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:31.981232 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:31.981201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:31.982293 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:31.982273 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:32.482548 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.482504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:32.571643 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.571610 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x"] Aug 04 12:59:32.575328 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.575307 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.577979 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.577958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fbddc9ad-serving-cert\"" Aug 04 12:59:32.587491 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.587464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x"] Aug 04 12:59:32.667308 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.667275 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:32.725542 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.725499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a0eeed-d6b3-424f-83fe-dd68b46f9275-openshift-service-ca\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.725727 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.725595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a0eeed-d6b3-424f-83fe-dd68b46f9275-maas-api-tls\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.725727 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.725716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zzc8\" (UniqueName: \"kubernetes.io/projected/75a0eeed-d6b3-424f-83fe-dd68b46f9275-kube-api-access-6zzc8\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.827000 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.826897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a0eeed-d6b3-424f-83fe-dd68b46f9275-maas-api-tls\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.827000 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.826995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zzc8\" (UniqueName: \"kubernetes.io/projected/75a0eeed-d6b3-424f-83fe-dd68b46f9275-kube-api-access-6zzc8\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.827304 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.827099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a0eeed-d6b3-424f-83fe-dd68b46f9275-openshift-service-ca\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.827742 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.827722 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a0eeed-d6b3-424f-83fe-dd68b46f9275-openshift-service-ca\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.829488 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.829470 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a0eeed-d6b3-424f-83fe-dd68b46f9275-maas-api-tls\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.836599 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.836575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zzc8\" (UniqueName: \"kubernetes.io/projected/75a0eeed-d6b3-424f-83fe-dd68b46f9275-kube-api-access-6zzc8\") pod \"maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:32.841466 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.841438 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:32.885716 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:32.885686 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:33.030486 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.030463 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x"] Aug 04 12:59:33.032824 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:59:33.032793 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a0eeed_d6b3_424f_83fe_dd68b46f9275.slice/crio-ece3cb471b830da14d768713b6761b994e2210e84b61e75105cb76ca3daa8410 WatchSource:0}: Error finding container ece3cb471b830da14d768713b6761b994e2210e84b61e75105cb76ca3daa8410: Status 404 returned error can't find the container with id ece3cb471b830da14d768713b6761b994e2210e84b61e75105cb76ca3daa8410 Aug 04 12:59:33.161870 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.161822 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:33.260476 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.260426 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 12:59:33.293023 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.292992 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk"] Aug 04 12:59:33.297980 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.297956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.300916 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.300892 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fbddc9ad\"" Aug 04 12:59:33.301041 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.300892 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fbddc9ad-dockercfg-bpn7n\"" Aug 04 12:59:33.309483 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.309452 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk"] Aug 04 12:59:33.433996 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.433899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7722b697-b0e7-4faf-876a-8a9db7b92f53-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.433996 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.433933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwg5g\" (UniqueName: \"kubernetes.io/projected/7722b697-b0e7-4faf-876a-8a9db7b92f53-kube-api-access-dwg5g\") pod \"payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.534756 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.534728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7722b697-b0e7-4faf-876a-8a9db7b92f53-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.534756 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.534762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwg5g\" (UniqueName: \"kubernetes.io/projected/7722b697-b0e7-4faf-876a-8a9db7b92f53-kube-api-access-dwg5g\") pod \"payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.535499 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.535474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7722b697-b0e7-4faf-876a-8a9db7b92f53-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.543584 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.543560 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwg5g\" (UniqueName: \"kubernetes.io/projected/7722b697-b0e7-4faf-876a-8a9db7b92f53-kube-api-access-dwg5g\") pod \"payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.610352 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.610317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:33.961463 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.961420 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk"] Aug 04 12:59:33.963729 ip-10-0-136-46 kubenswrapper[2578]: W0804 12:59:33.963700 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7722b697_b0e7_4faf_876a_8a9db7b92f53.slice/crio-3e734013d9493e803ac77e7e7ec4d14084e6030f3e2100b0eca9e8122d0b44d6 WatchSource:0}: Error finding container 3e734013d9493e803ac77e7e7ec4d14084e6030f3e2100b0eca9e8122d0b44d6: Status 404 returned error can't find the container with id 3e734013d9493e803ac77e7e7ec4d14084e6030f3e2100b0eca9e8122d0b44d6 Aug 04 12:59:33.994719 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.994688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" event={"ID":"7722b697-b0e7-4faf-876a-8a9db7b92f53","Type":"ContainerStarted","Data":"3e734013d9493e803ac77e7e7ec4d14084e6030f3e2100b0eca9e8122d0b44d6"} Aug 04 12:59:33.996281 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.996258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" event={"ID":"75a0eeed-d6b3-424f-83fe-dd68b46f9275","Type":"ContainerStarted","Data":"817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa"} Aug 04 12:59:33.996388 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:33.996296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" event={"ID":"75a0eeed-d6b3-424f-83fe-dd68b46f9275","Type":"ContainerStarted","Data":"ece3cb471b830da14d768713b6761b994e2210e84b61e75105cb76ca3daa8410"} Aug 04 12:59:34.018816 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:34.018768 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" podStartSLOduration=1.587147781 podStartE2EDuration="2.01874984s" totalImagesPullingTime="431.602059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:59:34.017325283 +0000 UTC m=+1904.972921082" watchObservedRunningTime="2026-08-04 12:59:34.01874984 +0000 UTC m=+1904.974345634" Aug 04 12:59:35.003682 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:35.003647 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" event={"ID":"7722b697-b0e7-4faf-876a-8a9db7b92f53","Type":"ContainerStarted","Data":"d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d"} Aug 04 12:59:35.004110 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:35.003716 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:35.004110 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:35.003867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:35.022816 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:35.022762 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" podStartSLOduration=2.022740025 podStartE2EDuration="2.022740025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 12:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 12:59:35.021298004 +0000 UTC m=+1905.976893799" watchObservedRunningTime="2026-08-04 12:59:35.022740025 +0000 UTC m=+1905.978335847" Aug 04 12:59:41.021566 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:41.021524 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 12:59:46.011588 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.011507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 12:59:46.077553 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.077518 2578 generic.go:356] "Generic (PLEG): container finished" podID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerID="631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce" exitCode=0 Aug 04 12:59:46.077733 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.077584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" event={"ID":"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce","Type":"ContainerDied","Data":"631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce"} Aug 04 12:59:46.078022 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.077996 2578 scope.go:117] "RemoveContainer" containerID="631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce" Aug 04 12:59:46.080136 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.080090 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 12:59:46.080229 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.080202 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 12:59:46.080267 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.080248 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 12:59:46.981248 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:46.981198 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:47.085348 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:47.085293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" event={"ID":"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce","Type":"ContainerStarted","Data":"bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee"} Aug 04 12:59:47.499525 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:47.499483 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:47.500851 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:47.500826 2578 patch_prober.go:28] interesting pod/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 04 12:59:47.500957 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:47.500885 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 04 12:59:48.500028 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:48.499983 2578 patch_prober.go:28] interesting pod/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 04 12:59:48.500564 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:48.500047 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 04 12:59:49.499248 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:49.499214 2578 patch_prober.go:28] interesting pod/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 04 12:59:49.499429 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:49.499275 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 04 12:59:50.499517 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:50.499476 2578 patch_prober.go:28] interesting pod/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 04 12:59:50.499906 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:50.499554 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 04 12:59:51.499883 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:51.499842 2578 patch_prober.go:28] interesting pod/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Aug 04 12:59:51.500348 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:51.499904 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Aug 04 12:59:52.503033 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:52.502997 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:52.503442 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:52.503393 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 12:59:52.504223 ip-10-0-136-46 kubenswrapper[2578]: I0804 12:59:52.504203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 13:00:03.273365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.273327 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x"] Aug 04 13:00:03.273876 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.273577 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" podUID="75a0eeed-d6b3-424f-83fe-dd68b46f9275" containerName="maas-api" containerID="cri-o://817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa" gracePeriod=30 Aug 04 13:00:03.367960 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.362452 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk"] Aug 04 13:00:03.367960 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.362699 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" podUID="7722b697-b0e7-4faf-876a-8a9db7b92f53" containerName="payload-processing" containerID="cri-o://d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d" gracePeriod=30 Aug 04 13:00:03.417973 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.417890 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:03.539116 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.539039 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 13:00:03.570220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.569312 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a0eeed-d6b3-424f-83fe-dd68b46f9275-openshift-service-ca\") pod \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " Aug 04 13:00:03.570220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.569437 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zzc8\" (UniqueName: \"kubernetes.io/projected/75a0eeed-d6b3-424f-83fe-dd68b46f9275-kube-api-access-6zzc8\") pod \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " Aug 04 13:00:03.570220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.569486 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a0eeed-d6b3-424f-83fe-dd68b46f9275-maas-api-tls\") pod \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\" (UID: \"75a0eeed-d6b3-424f-83fe-dd68b46f9275\") " Aug 04 13:00:03.570845 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.570799 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a0eeed-d6b3-424f-83fe-dd68b46f9275-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75a0eeed-d6b3-424f-83fe-dd68b46f9275" (UID: "75a0eeed-d6b3-424f-83fe-dd68b46f9275"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:00:03.574007 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.573978 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a0eeed-d6b3-424f-83fe-dd68b46f9275-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75a0eeed-d6b3-424f-83fe-dd68b46f9275" (UID: "75a0eeed-d6b3-424f-83fe-dd68b46f9275"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:00:03.574163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.574141 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a0eeed-d6b3-424f-83fe-dd68b46f9275-kube-api-access-6zzc8" (OuterVolumeSpecName: "kube-api-access-6zzc8") pod "75a0eeed-d6b3-424f-83fe-dd68b46f9275" (UID: "75a0eeed-d6b3-424f-83fe-dd68b46f9275"). InnerVolumeSpecName "kube-api-access-6zzc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:03.612901 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.612870 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 13:00:03.670293 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.670261 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7722b697-b0e7-4faf-876a-8a9db7b92f53-plugins-config-volume\") pod \"7722b697-b0e7-4faf-876a-8a9db7b92f53\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " Aug 04 13:00:03.670854 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.670827 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwg5g\" (UniqueName: \"kubernetes.io/projected/7722b697-b0e7-4faf-876a-8a9db7b92f53-kube-api-access-dwg5g\") pod \"7722b697-b0e7-4faf-876a-8a9db7b92f53\" (UID: \"7722b697-b0e7-4faf-876a-8a9db7b92f53\") " Aug 04 13:00:03.671110 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.670904 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7722b697-b0e7-4faf-876a-8a9db7b92f53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7722b697-b0e7-4faf-876a-8a9db7b92f53" (UID: "7722b697-b0e7-4faf-876a-8a9db7b92f53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:00:03.671966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.671942 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a0eeed-d6b3-424f-83fe-dd68b46f9275-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:03.672163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.672088 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a0eeed-d6b3-424f-83fe-dd68b46f9275-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:03.672163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.672156 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7722b697-b0e7-4faf-876a-8a9db7b92f53-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:03.672377 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.672172 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zzc8\" (UniqueName: \"kubernetes.io/projected/75a0eeed-d6b3-424f-83fe-dd68b46f9275-kube-api-access-6zzc8\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:03.673581 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.673544 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7722b697-b0e7-4faf-876a-8a9db7b92f53-kube-api-access-dwg5g" (OuterVolumeSpecName: "kube-api-access-dwg5g") pod "7722b697-b0e7-4faf-876a-8a9db7b92f53" (UID: "7722b697-b0e7-4faf-876a-8a9db7b92f53"). InnerVolumeSpecName "kube-api-access-dwg5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:03.714484 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.713285 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r"] Aug 04 13:00:03.714484 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.713531 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" containerID="cri-o://bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee" gracePeriod=30 Aug 04 13:00:03.773603 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.773573 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwg5g\" (UniqueName: \"kubernetes.io/projected/7722b697-b0e7-4faf-876a-8a9db7b92f53-kube-api-access-dwg5g\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:03.975670 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:03.975633 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:04.041099 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.041063 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:04.178566 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.178531 2578 generic.go:356] "Generic (PLEG): container finished" podID="75a0eeed-d6b3-424f-83fe-dd68b46f9275" containerID="817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa" exitCode=0 Aug 04 13:00:04.178771 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.178581 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" Aug 04 13:00:04.178771 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.178630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" event={"ID":"75a0eeed-d6b3-424f-83fe-dd68b46f9275","Type":"ContainerDied","Data":"817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa"} Aug 04 13:00:04.178771 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.178692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x" event={"ID":"75a0eeed-d6b3-424f-83fe-dd68b46f9275","Type":"ContainerDied","Data":"ece3cb471b830da14d768713b6761b994e2210e84b61e75105cb76ca3daa8410"} Aug 04 13:00:04.178771 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.178724 2578 scope.go:117] "RemoveContainer" containerID="817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa" Aug 04 13:00:04.180041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.180014 2578 generic.go:356] "Generic (PLEG): container finished" podID="7722b697-b0e7-4faf-876a-8a9db7b92f53" containerID="d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d" exitCode=0 Aug 04 13:00:04.180221 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.180052 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" event={"ID":"7722b697-b0e7-4faf-876a-8a9db7b92f53","Type":"ContainerDied","Data":"d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d"} Aug 04 13:00:04.180221 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.180069 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" Aug 04 13:00:04.180221 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.180106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk" event={"ID":"7722b697-b0e7-4faf-876a-8a9db7b92f53","Type":"ContainerDied","Data":"3e734013d9493e803ac77e7e7ec4d14084e6030f3e2100b0eca9e8122d0b44d6"} Aug 04 13:00:04.189182 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.189163 2578 scope.go:117] "RemoveContainer" containerID="817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa" Aug 04 13:00:04.189511 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:04.189487 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa\": container with ID starting with 817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa not found: ID does not exist" containerID="817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa" Aug 04 13:00:04.189603 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.189522 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa"} err="failed to get container status \"817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa\": rpc error: code = NotFound desc = could not find container \"817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa\": container with ID starting with 817321fbc6edd0eeaee543b42da1b2852d41c9f286741259d78ab15b62e727fa not found: ID does not exist" Aug 04 13:00:04.189603 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.189545 2578 scope.go:117] "RemoveContainer" containerID="d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d" Aug 04 13:00:04.197899 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.197881 2578 scope.go:117] "RemoveContainer" containerID="d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d" Aug 04 13:00:04.198183 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:04.198156 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d\": container with ID starting with d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d not found: ID does not exist" containerID="d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d" Aug 04 13:00:04.198280 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.198185 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d"} err="failed to get container status \"d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d\": rpc error: code = NotFound desc = could not find container \"d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d\": container with ID starting with d34faf194a89345765e62ee602d9f80bf4ee6731cab93074bd83248844e6632d not found: ID does not exist" Aug 04 13:00:04.206088 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.206065 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x"] Aug 04 13:00:04.208779 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.208757 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fbddc9ad-787f9cb96b-s499x"] Aug 04 13:00:04.220887 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.220864 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk"] Aug 04 13:00:04.227601 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.227539 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fbddc9ad-c5bb4fb5-fpkvk"] Aug 04 13:00:04.244916 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.244888 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:04.538943 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.538855 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd"] Aug 04 13:00:04.539356 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.539272 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a0eeed-d6b3-424f-83fe-dd68b46f9275" containerName="maas-api" Aug 04 13:00:04.539356 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.539284 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a0eeed-d6b3-424f-83fe-dd68b46f9275" containerName="maas-api" Aug 04 13:00:04.539356 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.539297 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7722b697-b0e7-4faf-876a-8a9db7b92f53" containerName="payload-processing" Aug 04 13:00:04.539356 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.539305 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7722b697-b0e7-4faf-876a-8a9db7b92f53" containerName="payload-processing" Aug 04 13:00:04.539516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.539360 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a0eeed-d6b3-424f-83fe-dd68b46f9275" containerName="maas-api" Aug 04 13:00:04.539516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.539373 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7722b697-b0e7-4faf-876a-8a9db7b92f53" containerName="payload-processing" Aug 04 13:00:04.544233 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.544215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.548867 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.548847 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f143cf24-openshift-default-dockercfg-tjf6k\"" Aug 04 13:00:04.561904 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.561869 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd"] Aug 04 13:00:04.579459 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-data\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.579640 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-certs\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.579640 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-credential-socket\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.579640 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istiod-ca-cert\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.579640 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-envoy\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.579640 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-podinfo\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.580006 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-token\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.580006 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-socket\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.580006 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.579723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsmh\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-kube-api-access-vnsmh\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681011 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.680972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-socket\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681011 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsmh\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-kube-api-access-vnsmh\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-data\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-certs\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-credential-socket\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681206 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istiod-ca-cert\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-envoy\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-podinfo\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-token\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681688 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-credential-socket\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681688 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681644 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-data\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681703 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-certs\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.681869 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.681849 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-socket\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.682249 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.682192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istiod-ca-cert\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.683819 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.683793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-envoy\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.684005 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.683985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-podinfo\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.694403 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.694375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsmh\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-kube-api-access-vnsmh\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.695513 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.695487 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-token\") pod \"e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.751712 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.751678 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:04.851918 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.851881 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:04.854897 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.854871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:04.999721 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:04.999688 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd"] Aug 04 13:00:05.003252 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:00:05.003223 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b6358a_9dfe_427c_a46f_959f5c0528bf.slice/crio-900f0d21e7f50f27e625e121feb23184f9bbc38f88c1ebd0120d4ef02aeab86f WatchSource:0}: Error finding container 900f0d21e7f50f27e625e121feb23184f9bbc38f88c1ebd0120d4ef02aeab86f: Status 404 returned error can't find the container with id 900f0d21e7f50f27e625e121feb23184f9bbc38f88c1ebd0120d4ef02aeab86f Aug 04 13:00:05.005366 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.005334 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:05.005469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.005413 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:05.005469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.005457 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:05.189735 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.189636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" event={"ID":"e9b6358a-9dfe-427c-a46f-959f5c0528bf","Type":"ContainerStarted","Data":"b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a"} Aug 04 13:00:05.189735 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.189695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" event={"ID":"e9b6358a-9dfe-427c-a46f-959f5c0528bf","Type":"ContainerStarted","Data":"900f0d21e7f50f27e625e121feb23184f9bbc38f88c1ebd0120d4ef02aeab86f"} Aug 04 13:00:05.213875 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.213762 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podStartSLOduration=1.213743867 podStartE2EDuration="1.213743867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:00:05.212174492 +0000 UTC m=+1936.167770287" watchObservedRunningTime="2026-08-04 13:00:05.213743867 +0000 UTC m=+1936.169339660" Aug 04 13:00:05.552978 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.552893 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a0eeed-d6b3-424f-83fe-dd68b46f9275" path="/var/lib/kubelet/pods/75a0eeed-d6b3-424f-83fe-dd68b46f9275/volumes" Aug 04 13:00:05.553355 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.553281 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7722b697-b0e7-4faf-876a-8a9db7b92f53" path="/var/lib/kubelet/pods/7722b697-b0e7-4faf-876a-8a9db7b92f53/volumes" Aug 04 13:00:05.855565 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:05.855535 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:06.563151 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:06.563092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:06.660710 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:06.660676 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:06.860141 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:06.860099 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:07.201454 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.201373 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:07.202335 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.202318 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:07.505896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.505811 2578 patch_prober.go:28] interesting pod/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 13:00:07.505896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.505870 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 13:00:07.664702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.664665 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z"] Aug 04 13:00:07.670379 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.670352 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.673099 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.673073 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f143cf24-serving-cert\"" Aug 04 13:00:07.678469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.678433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z"] Aug 04 13:00:07.696795 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.696768 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:07.711070 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.711041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55605396-e163-4c8c-91d5-706eb333f2fe-openshift-service-ca\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.711269 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.711188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55605396-e163-4c8c-91d5-706eb333f2fe-maas-api-tls\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.711269 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.711238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gxlx\" (UniqueName: \"kubernetes.io/projected/55605396-e163-4c8c-91d5-706eb333f2fe-kube-api-access-4gxlx\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.812771 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.812693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55605396-e163-4c8c-91d5-706eb333f2fe-openshift-service-ca\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.812924 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.812805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55605396-e163-4c8c-91d5-706eb333f2fe-maas-api-tls\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.812924 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.812833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gxlx\" (UniqueName: \"kubernetes.io/projected/55605396-e163-4c8c-91d5-706eb333f2fe-kube-api-access-4gxlx\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.813382 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.813356 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55605396-e163-4c8c-91d5-706eb333f2fe-openshift-service-ca\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.815253 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.815232 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55605396-e163-4c8c-91d5-706eb333f2fe-maas-api-tls\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.824554 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.824525 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gxlx\" (UniqueName: \"kubernetes.io/projected/55605396-e163-4c8c-91d5-706eb333f2fe-kube-api-access-4gxlx\") pod \"maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:07.984744 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:07.984702 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:08.132902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.132879 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z"] Aug 04 13:00:08.134832 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:00:08.134805 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55605396_e163_4c8c_91d5_706eb333f2fe.slice/crio-d7159d58b03bda21438ad1c79c467ec4cceebb1e2cbae4e6ab95665af1d9d564 WatchSource:0}: Error finding container d7159d58b03bda21438ad1c79c467ec4cceebb1e2cbae4e6ab95665af1d9d564: Status 404 returned error can't find the container with id d7159d58b03bda21438ad1c79c467ec4cceebb1e2cbae4e6ab95665af1d9d564 Aug 04 13:00:08.208091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.208058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" event={"ID":"55605396-e163-4c8c-91d5-706eb333f2fe","Type":"ContainerStarted","Data":"d7159d58b03bda21438ad1c79c467ec4cceebb1e2cbae4e6ab95665af1d9d564"} Aug 04 13:00:08.224061 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.224027 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:08.415640 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.415599 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg"] Aug 04 13:00:08.420578 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.420550 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.423313 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.423277 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f143cf24-dockercfg-pbpmv\"" Aug 04 13:00:08.423313 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.423279 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f143cf24\"" Aug 04 13:00:08.432221 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.432191 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg"] Aug 04 13:00:08.520169 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.520095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.520341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.520187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptc44\" (UniqueName: \"kubernetes.io/projected/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-kube-api-access-ptc44\") pod \"payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.621592 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.621553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.621592 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.621592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptc44\" (UniqueName: \"kubernetes.io/projected/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-kube-api-access-ptc44\") pod \"payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.622444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.622421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.622725 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.622707 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:08.630893 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.630868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptc44\" (UniqueName: \"kubernetes.io/projected/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-kube-api-access-ptc44\") pod \"payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.737894 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.737869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:08.895954 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.895925 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg"] Aug 04 13:00:08.903253 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:00:08.903171 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e90385_6d63_4c23_b4ef_5f65f47ac8ce.slice/crio-fa379845b70841ae31c5e54ac136556eb584ccfb3628057bcf4728d9d38dc94a WatchSource:0}: Error finding container fa379845b70841ae31c5e54ac136556eb584ccfb3628057bcf4728d9d38dc94a: Status 404 returned error can't find the container with id fa379845b70841ae31c5e54ac136556eb584ccfb3628057bcf4728d9d38dc94a Aug 04 13:00:08.994389 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:08.994363 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 13:00:09.027297 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027271 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-socket\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027297 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027302 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-data\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027319 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-token\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-certs\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027385 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istiod-ca-cert\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027418 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-podinfo\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027459 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-envoy\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027487 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm28t\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-kube-api-access-tm28t\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027517 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-credential-socket\") pod \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\" (UID: \"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce\") " Aug 04 13:00:09.027966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027692 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:09.027966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027727 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:09.027966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027743 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:09.027966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027954 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.028207 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027973 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.028207 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.027985 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.028575 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.028402 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:00:09.028575 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.028531 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:09.030893 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.030868 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:00:09.031033 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.031013 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:09.031085 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.031016 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-kube-api-access-tm28t" (OuterVolumeSpecName: "kube-api-access-tm28t") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "kube-api-access-tm28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:09.031085 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.031056 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" (UID: "742809a6-9e9f-4cd5-8c90-905ce4c0e5ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:09.129411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.129375 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.129411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.129403 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.129411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.129412 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.129411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.129420 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.129682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.129429 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm28t\" (UniqueName: \"kubernetes.io/projected/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-kube-api-access-tm28t\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.129682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.129442 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:09.216012 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.215979 2578 generic.go:356] "Generic (PLEG): container finished" podID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerID="bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee" exitCode=0 Aug 04 13:00:09.216210 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.216035 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" Aug 04 13:00:09.216210 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.216062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" event={"ID":"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce","Type":"ContainerDied","Data":"bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee"} Aug 04 13:00:09.216210 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.216150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r" event={"ID":"742809a6-9e9f-4cd5-8c90-905ce4c0e5ce","Type":"ContainerDied","Data":"a148e5784903b3b4c45e1f58bb64d890e0dc6ab9dffcc503c03982febaa27dfa"} Aug 04 13:00:09.216210 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.216157 2578 scope.go:117] "RemoveContainer" containerID="bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee" Aug 04 13:00:09.218744 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.218719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" event={"ID":"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce","Type":"ContainerStarted","Data":"3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75"} Aug 04 13:00:09.218870 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.218778 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:09.218870 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.218804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" event={"ID":"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce","Type":"ContainerStarted","Data":"fa379845b70841ae31c5e54ac136556eb584ccfb3628057bcf4728d9d38dc94a"} Aug 04 13:00:09.220764 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.220702 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" event={"ID":"55605396-e163-4c8c-91d5-706eb333f2fe","Type":"ContainerStarted","Data":"b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171"} Aug 04 13:00:09.220949 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.220928 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:09.228624 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.228601 2578 scope.go:117] "RemoveContainer" containerID="631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce" Aug 04 13:00:09.238068 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.238048 2578 scope.go:117] "RemoveContainer" containerID="bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee" Aug 04 13:00:09.238484 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:09.238449 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee\": container with ID starting with bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee not found: ID does not exist" containerID="bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee" Aug 04 13:00:09.238637 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.238485 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee"} err="failed to get container status \"bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee\": rpc error: code = NotFound desc = could not find container \"bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee\": container with ID starting with bdbf7f0233464bfb9c323b39fedf3178eeb93b04f4e5723e075dc16eebb437ee not found: ID does not exist" Aug 04 13:00:09.238637 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.238512 2578 scope.go:117] "RemoveContainer" containerID="631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce" Aug 04 13:00:09.239374 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:09.238972 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce\": container with ID starting with 631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce not found: ID does not exist" containerID="631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce" Aug 04 13:00:09.239374 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.239345 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce"} err="failed to get container status \"631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce\": rpc error: code = NotFound desc = could not find container \"631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce\": container with ID starting with 631cc775f54cc5c716ae7dc85d288be7858b352629638cbb78a46a32ea990bce not found: ID does not exist" Aug 04 13:00:09.240045 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.240005 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" podStartSLOduration=1.239991458 podStartE2EDuration="1.239991458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:00:09.239115021 +0000 UTC m=+1940.194710816" watchObservedRunningTime="2026-08-04 13:00:09.239991458 +0000 UTC m=+1940.195587229" Aug 04 13:00:09.259400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.259349 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" podStartSLOduration=1.414526211 podStartE2EDuration="2.259331382s" totalImagesPullingTime="844.805171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:00:09.257157943 +0000 UTC m=+1940.212753739" watchObservedRunningTime="2026-08-04 13:00:09.259331382 +0000 UTC m=+1940.214927175" Aug 04 13:00:09.273285 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.273253 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r"] Aug 04 13:00:09.276182 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.276157 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fbddc9ad-gw-openshift-default-b4769d776-krf2r"] Aug 04 13:00:09.553486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:09.553403 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" path="/var/lib/kubelet/pods/742809a6-9e9f-4cd5-8c90-905ce4c0e5ce/volumes" Aug 04 13:00:11.342735 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:11.342701 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:15.234145 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:15.234087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:20.230064 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.229978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:20.283800 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.283768 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerID="b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a" exitCode=0 Aug 04 13:00:20.283984 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.283841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" event={"ID":"e9b6358a-9dfe-427c-a46f-959f5c0528bf","Type":"ContainerDied","Data":"b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a"} Aug 04 13:00:20.284303 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.284284 2578 scope.go:117] "RemoveContainer" containerID="b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a" Aug 04 13:00:20.285461 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.285425 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:20.285542 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.285525 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:20.285617 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:20.285601 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:21.292354 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:21.292313 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" event={"ID":"e9b6358a-9dfe-427c-a46f-959f5c0528bf","Type":"ContainerStarted","Data":"dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06"} Aug 04 13:00:21.855225 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:21.855174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:21.856378 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:21.856352 2578 patch_prober.go:28] interesting pod/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 04 13:00:21.856521 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:21.856409 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 04 13:00:22.201931 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:22.201823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:22.856151 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:22.856092 2578 patch_prober.go:28] interesting pod/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 04 13:00:22.856543 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:22.856175 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 04 13:00:23.856043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:23.856008 2578 patch_prober.go:28] interesting pod/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 04 13:00:23.856248 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:23.856065 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 04 13:00:24.855909 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:24.855877 2578 patch_prober.go:28] interesting pod/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 04 13:00:24.856108 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:24.855935 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 04 13:00:25.856018 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:25.855983 2578 patch_prober.go:28] interesting pod/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Aug 04 13:00:25.856431 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:25.856046 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Aug 04 13:00:26.859286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:26.859250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:26.860314 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:26.860297 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:38.449414 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.447046 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z"] Aug 04 13:00:38.449414 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.447400 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" podUID="55605396-e163-4c8c-91d5-706eb333f2fe" containerName="maas-api" containerID="cri-o://b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171" gracePeriod=30 Aug 04 13:00:38.459045 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.458973 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:38.508724 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.508382 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg"] Aug 04 13:00:38.508724 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.508670 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" podUID="c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" containerName="payload-pre-processing" containerID="cri-o://3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75" gracePeriod=30 Aug 04 13:00:38.558648 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.558600 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:38.774307 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.774281 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:38.797397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.797365 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:38.848346 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848316 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55605396-e163-4c8c-91d5-706eb333f2fe-openshift-service-ca\") pod \"55605396-e163-4c8c-91d5-706eb333f2fe\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " Aug 04 13:00:38.848346 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848350 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55605396-e163-4c8c-91d5-706eb333f2fe-maas-api-tls\") pod \"55605396-e163-4c8c-91d5-706eb333f2fe\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " Aug 04 13:00:38.848600 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848381 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-plugins-config-volume\") pod \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " Aug 04 13:00:38.848600 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848408 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gxlx\" (UniqueName: \"kubernetes.io/projected/55605396-e163-4c8c-91d5-706eb333f2fe-kube-api-access-4gxlx\") pod \"55605396-e163-4c8c-91d5-706eb333f2fe\" (UID: \"55605396-e163-4c8c-91d5-706eb333f2fe\") " Aug 04 13:00:38.848600 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptc44\" (UniqueName: \"kubernetes.io/projected/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-kube-api-access-ptc44\") pod \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\" (UID: \"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce\") " Aug 04 13:00:38.848770 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848667 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55605396-e163-4c8c-91d5-706eb333f2fe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55605396-e163-4c8c-91d5-706eb333f2fe" (UID: "55605396-e163-4c8c-91d5-706eb333f2fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:00:38.848857 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848820 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" (UID: "c1e90385-6d63-4c23-b4ef-5f65f47ac8ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:00:38.848987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.848904 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55605396-e163-4c8c-91d5-706eb333f2fe-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:38.850722 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.850698 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55605396-e163-4c8c-91d5-706eb333f2fe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55605396-e163-4c8c-91d5-706eb333f2fe" (UID: "55605396-e163-4c8c-91d5-706eb333f2fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:00:38.850722 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.850706 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55605396-e163-4c8c-91d5-706eb333f2fe-kube-api-access-4gxlx" (OuterVolumeSpecName: "kube-api-access-4gxlx") pod "55605396-e163-4c8c-91d5-706eb333f2fe" (UID: "55605396-e163-4c8c-91d5-706eb333f2fe"). InnerVolumeSpecName "kube-api-access-4gxlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:38.850856 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.850766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-kube-api-access-ptc44" (OuterVolumeSpecName: "kube-api-access-ptc44") pod "c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" (UID: "c1e90385-6d63-4c23-b4ef-5f65f47ac8ce"). InnerVolumeSpecName "kube-api-access-ptc44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:38.933619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.933581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:38.949983 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.949952 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55605396-e163-4c8c-91d5-706eb333f2fe-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:38.949983 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.949982 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:38.950202 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.949992 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gxlx\" (UniqueName: \"kubernetes.io/projected/55605396-e163-4c8c-91d5-706eb333f2fe-kube-api-access-4gxlx\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:38.950202 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.950000 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptc44\" (UniqueName: \"kubernetes.io/projected/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce-kube-api-access-ptc44\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:38.994801 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:38.994683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:39.392408 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.392375 2578 generic.go:356] "Generic (PLEG): container finished" podID="55605396-e163-4c8c-91d5-706eb333f2fe" containerID="b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171" exitCode=0 Aug 04 13:00:39.392587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.392431 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" Aug 04 13:00:39.392587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.392459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" event={"ID":"55605396-e163-4c8c-91d5-706eb333f2fe","Type":"ContainerDied","Data":"b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171"} Aug 04 13:00:39.392587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.392512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z" event={"ID":"55605396-e163-4c8c-91d5-706eb333f2fe","Type":"ContainerDied","Data":"d7159d58b03bda21438ad1c79c467ec4cceebb1e2cbae4e6ab95665af1d9d564"} Aug 04 13:00:39.392587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.392543 2578 scope.go:117] "RemoveContainer" containerID="b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171" Aug 04 13:00:39.393813 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.393787 2578 generic.go:356] "Generic (PLEG): container finished" podID="c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" containerID="3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75" exitCode=0 Aug 04 13:00:39.393955 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.393856 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" Aug 04 13:00:39.393955 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.393875 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" event={"ID":"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce","Type":"ContainerDied","Data":"3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75"} Aug 04 13:00:39.393955 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.393914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg" event={"ID":"c1e90385-6d63-4c23-b4ef-5f65f47ac8ce","Type":"ContainerDied","Data":"fa379845b70841ae31c5e54ac136556eb584ccfb3628057bcf4728d9d38dc94a"} Aug 04 13:00:39.403194 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.403173 2578 scope.go:117] "RemoveContainer" containerID="b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171" Aug 04 13:00:39.403542 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:39.403516 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171\": container with ID starting with b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171 not found: ID does not exist" containerID="b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171" Aug 04 13:00:39.403636 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.403565 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171"} err="failed to get container status \"b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171\": rpc error: code = NotFound desc = could not find container \"b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171\": container with ID starting with b28699720ee3012cbfd377f6d33573363ac09e59d85284c755d689079763e171 not found: ID does not exist" Aug 04 13:00:39.403636 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.403590 2578 scope.go:117] "RemoveContainer" containerID="3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75" Aug 04 13:00:39.411697 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.411679 2578 scope.go:117] "RemoveContainer" containerID="3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75" Aug 04 13:00:39.411958 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:39.411944 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75\": container with ID starting with 3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75 not found: ID does not exist" containerID="3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75" Aug 04 13:00:39.411998 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.411965 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75"} err="failed to get container status \"3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75\": rpc error: code = NotFound desc = could not find container \"3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75\": container with ID starting with 3abdc69b816c0c15373894b2098f6bc8ffecfc83551bfcbbfd13a71fe324ce75 not found: ID does not exist" Aug 04 13:00:39.419311 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.419289 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z"] Aug 04 13:00:39.423671 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.423651 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f143cf24-6f96f7b94c-v522z"] Aug 04 13:00:39.436400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.436381 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg"] Aug 04 13:00:39.440941 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.440922 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f143cf24-646875d44-gkzdg"] Aug 04 13:00:39.553187 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.553157 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55605396-e163-4c8c-91d5-706eb333f2fe" path="/var/lib/kubelet/pods/55605396-e163-4c8c-91d5-706eb333f2fe/volumes" Aug 04 13:00:39.553553 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.553511 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" path="/var/lib/kubelet/pods/c1e90385-6d63-4c23-b4ef-5f65f47ac8ce/volumes" Aug 04 13:00:39.843563 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.843530 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd"] Aug 04 13:00:39.843809 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.843784 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" containerID="cri-o://dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06" gracePeriod=30 Aug 04 13:00:39.867359 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:39.867330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:40.081998 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.081921 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:40.315995 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.315902 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:40.613419 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.613388 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8"] Aug 04 13:00:40.613975 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.613955 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" containerName="payload-pre-processing" Aug 04 13:00:40.614066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.613978 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" containerName="payload-pre-processing" Aug 04 13:00:40.614066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614004 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" Aug 04 13:00:40.614066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614013 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" Aug 04 13:00:40.614066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614031 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" Aug 04 13:00:40.614066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614040 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" Aug 04 13:00:40.614066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614066 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55605396-e163-4c8c-91d5-706eb333f2fe" containerName="maas-api" Aug 04 13:00:40.614411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614074 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="55605396-e163-4c8c-91d5-706eb333f2fe" containerName="maas-api" Aug 04 13:00:40.614411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614200 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" Aug 04 13:00:40.614411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614215 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="55605396-e163-4c8c-91d5-706eb333f2fe" containerName="maas-api" Aug 04 13:00:40.614411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.614230 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e90385-6d63-4c23-b4ef-5f65f47ac8ce" containerName="payload-pre-processing" Aug 04 13:00:40.618912 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.618892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.621702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.621682 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cb728584-openshift-default-dockercfg-w5j7s\"" Aug 04 13:00:40.628301 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.628274 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8"] Aug 04 13:00:40.632582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.632558 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:40.665820 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.665785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6p8\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-kube-api-access-dm6p8\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666016 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.665835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54452d78-0bbd-4181-afd3-382b44096684-istio-podinfo\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666016 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.665898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-istio-token\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666016 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.665923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-data\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666016 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.665952 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54452d78-0bbd-4181-afd3-382b44096684-istiod-ca-cert\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666016 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.665992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-credential-socket\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666331 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.666060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-socket\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666331 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.666188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-envoy\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.666331 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.666228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-certs\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767236 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6p8\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-kube-api-access-dm6p8\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767236 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54452d78-0bbd-4181-afd3-382b44096684-istio-podinfo\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-istio-token\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-data\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54452d78-0bbd-4181-afd3-382b44096684-istiod-ca-cert\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-credential-socket\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-socket\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-envoy\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767887 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-certs\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767887 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767808 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-data\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.767887 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767852 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-socket\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.768054 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.767886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-credential-socket\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.768054 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.768026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-certs\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.768151 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.768064 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54452d78-0bbd-4181-afd3-382b44096684-istiod-ca-cert\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.769815 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.769794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-envoy\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.769977 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.769955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54452d78-0bbd-4181-afd3-382b44096684-istio-podinfo\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.777113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.777083 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-istio-token\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.777113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.777092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6p8\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-kube-api-access-dm6p8\") pod \"e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:40.883590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.883500 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:40.930598 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:40.930562 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:41.080386 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.080356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8"] Aug 04 13:00:41.083088 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:00:41.083057 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54452d78_0bbd_4181_afd3_382b44096684.slice/crio-ce4e41e76a7a0db731234ec6024dab76635b7b5e1ab83da2a603e0e765961b19 WatchSource:0}: Error finding container ce4e41e76a7a0db731234ec6024dab76635b7b5e1ab83da2a603e0e765961b19: Status 404 returned error can't find the container with id ce4e41e76a7a0db731234ec6024dab76635b7b5e1ab83da2a603e0e765961b19 Aug 04 13:00:41.085482 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.085447 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:41.085547 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.085533 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:41.085598 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.085580 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:41.408651 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.408557 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" event={"ID":"54452d78-0bbd-4181-afd3-382b44096684","Type":"ContainerStarted","Data":"aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832"} Aug 04 13:00:41.408651 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.408633 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" event={"ID":"54452d78-0bbd-4181-afd3-382b44096684","Type":"ContainerStarted","Data":"ce4e41e76a7a0db731234ec6024dab76635b7b5e1ab83da2a603e0e765961b19"} Aug 04 13:00:41.433078 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.433011 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podStartSLOduration=1.432994432 podStartE2EDuration="1.432994432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:00:41.430296239 +0000 UTC m=+1972.385892034" watchObservedRunningTime="2026-08-04 13:00:41.432994432 +0000 UTC m=+1972.388590225" Aug 04 13:00:41.861746 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.861709 2578 patch_prober.go:28] interesting pod/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 13:00:41.862220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.861779 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 13:00:41.931150 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:41.931082 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:42.773765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:42.773733 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:42.873900 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:42.873870 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:42.936100 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:42.936069 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:43.418617 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.418580 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:43.419542 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.419522 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:43.880234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.880201 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh"] Aug 04 13:00:43.880935 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.880907 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="742809a6-9e9f-4cd5-8c90-905ce4c0e5ce" containerName="istio-proxy" Aug 04 13:00:43.883658 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.883640 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:43.886675 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.886655 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cb728584-serving-cert\"" Aug 04 13:00:43.903682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.903655 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh"] Aug 04 13:00:43.924302 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:43.924255 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:44.003697 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.003661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.003881 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.003790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9dw\" (UniqueName: \"kubernetes.io/projected/0ac84152-a75a-4099-bb84-33f309059333-kube-api-access-gv9dw\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.003881 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.003866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac84152-a75a-4099-bb84-33f309059333-openshift-service-ca\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.104966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.104927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9dw\" (UniqueName: \"kubernetes.io/projected/0ac84152-a75a-4099-bb84-33f309059333-kube-api-access-gv9dw\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.105205 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.105017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac84152-a75a-4099-bb84-33f309059333-openshift-service-ca\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.105205 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.105059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.105308 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:44.105242 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-serving-cert: secret "maas-api-e2e-derive-cb728584-serving-cert" not found Aug 04 13:00:44.105344 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:44.105307 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls podName:0ac84152-a75a-4099-bb84-33f309059333 nodeName:}" failed. No retries permitted until 2026-08-04 13:00:44.605288355 +0000 UTC m=+1975.560884140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls") pod "maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" (UID: "0ac84152-a75a-4099-bb84-33f309059333") : secret "maas-api-e2e-derive-cb728584-serving-cert" not found Aug 04 13:00:44.105657 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.105638 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac84152-a75a-4099-bb84-33f309059333-openshift-service-ca\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.115837 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.115811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9dw\" (UniqueName: \"kubernetes.io/projected/0ac84152-a75a-4099-bb84-33f309059333-kube-api-access-gv9dw\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.240199 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.240080 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:44.546781 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.546695 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:44.609583 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.609549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.612213 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.612187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls\") pod \"maas-api-e2e-derive-cb728584-595dd6986b-q4xbh\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.649073 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.649037 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:00:44.805463 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.805370 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:44.952936 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:44.952908 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh"] Aug 04 13:00:44.954437 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:00:44.954410 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac84152_a75a_4099_bb84_33f309059333.slice/crio-ab3e2ccc964eebf7a1f844bb1e459c15807a95f011f3ebb2cd4caf0b7ec6b42e WatchSource:0}: Error finding container ab3e2ccc964eebf7a1f844bb1e459c15807a95f011f3ebb2cd4caf0b7ec6b42e: Status 404 returned error can't find the container with id ab3e2ccc964eebf7a1f844bb1e459c15807a95f011f3ebb2cd4caf0b7ec6b42e Aug 04 13:00:45.087801 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.087776 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:45.216232 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216195 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-data\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216251 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-socket\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216307 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-certs\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216350 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istiod-ca-cert\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-podinfo\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216611 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216448 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-token\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216611 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-credential-socket\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216611 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216527 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-envoy\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216611 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnsmh\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-kube-api-access-vnsmh\") pod \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\" (UID: \"e9b6358a-9dfe-427c-a46f-959f5c0528bf\") " Aug 04 13:00:45.216611 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216540 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:45.216611 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216588 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:45.216925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216677 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:45.216925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216788 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:45.216925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.216836 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:00:45.217066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.217041 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.217066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.217063 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.217161 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.217078 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.217161 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.217090 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.217161 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.217103 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.218905 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.218876 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:00:45.219258 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.218943 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:00:45.219258 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.219208 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:45.219357 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.219322 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-kube-api-access-vnsmh" (OuterVolumeSpecName: "kube-api-access-vnsmh") pod "e9b6358a-9dfe-427c-a46f-959f5c0528bf" (UID: "e9b6358a-9dfe-427c-a46f-959f5c0528bf"). InnerVolumeSpecName "kube-api-access-vnsmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:00:45.318180 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.318146 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.318180 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.318175 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.318180 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.318186 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b6358a-9dfe-427c-a46f-959f5c0528bf-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.318408 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.318198 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsmh\" (UniqueName: \"kubernetes.io/projected/e9b6358a-9dfe-427c-a46f-959f5c0528bf-kube-api-access-vnsmh\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:00:45.430872 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.430841 2578 generic.go:356] "Generic (PLEG): container finished" podID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerID="dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06" exitCode=0 Aug 04 13:00:45.431041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.430890 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" Aug 04 13:00:45.431041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.430930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" event={"ID":"e9b6358a-9dfe-427c-a46f-959f5c0528bf","Type":"ContainerDied","Data":"dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06"} Aug 04 13:00:45.431041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.430985 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd" event={"ID":"e9b6358a-9dfe-427c-a46f-959f5c0528bf","Type":"ContainerDied","Data":"900f0d21e7f50f27e625e121feb23184f9bbc38f88c1ebd0120d4ef02aeab86f"} Aug 04 13:00:45.431041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.431012 2578 scope.go:117] "RemoveContainer" containerID="dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06" Aug 04 13:00:45.432732 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.432701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" event={"ID":"0ac84152-a75a-4099-bb84-33f309059333","Type":"ContainerStarted","Data":"83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04"} Aug 04 13:00:45.432863 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.432753 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" event={"ID":"0ac84152-a75a-4099-bb84-33f309059333","Type":"ContainerStarted","Data":"ab3e2ccc964eebf7a1f844bb1e459c15807a95f011f3ebb2cd4caf0b7ec6b42e"} Aug 04 13:00:45.440535 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.440513 2578 scope.go:117] "RemoveContainer" containerID="b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a" Aug 04 13:00:45.448839 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.448822 2578 scope.go:117] "RemoveContainer" containerID="dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06" Aug 04 13:00:45.449150 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:45.449106 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06\": container with ID starting with dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06 not found: ID does not exist" containerID="dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06" Aug 04 13:00:45.449211 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.449162 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06"} err="failed to get container status \"dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06\": rpc error: code = NotFound desc = could not find container \"dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06\": container with ID starting with dd45aaa6d73aa25692ab57df818bb8d251ed0cf483de435abcf180c431ef7a06 not found: ID does not exist" Aug 04 13:00:45.449211 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.449179 2578 scope.go:117] "RemoveContainer" containerID="b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a" Aug 04 13:00:45.449393 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:00:45.449375 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a\": container with ID starting with b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a not found: ID does not exist" containerID="b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a" Aug 04 13:00:45.449435 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.449396 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a"} err="failed to get container status \"b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a\": rpc error: code = NotFound desc = could not find container \"b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a\": container with ID starting with b71444e1409f8e186d3ba29c27ddb117b2781a247d9446b2af7f64927102433a not found: ID does not exist" Aug 04 13:00:45.458295 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.458249 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" podStartSLOduration=2.088051743 podStartE2EDuration="2.458235389s" totalImagesPullingTime="370.183646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:00:45.456373046 +0000 UTC m=+1976.411968842" watchObservedRunningTime="2026-08-04 13:00:45.458235389 +0000 UTC m=+1976.413831182" Aug 04 13:00:45.472390 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.472364 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd"] Aug 04 13:00:45.479764 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.479739 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f143cf24-openshift-default-77b7b6c48b-nsdfd"] Aug 04 13:00:45.553615 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:45.553548 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" path="/var/lib/kubelet/pods/e9b6358a-9dfe-427c-a46f-959f5c0528bf/volumes" Aug 04 13:00:46.439017 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:46.438980 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:52.450135 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:52.450085 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:00:56.493431 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:56.493398 2578 generic.go:356] "Generic (PLEG): container finished" podID="54452d78-0bbd-4181-afd3-382b44096684" containerID="aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832" exitCode=0 Aug 04 13:00:56.493848 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:56.493485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" event={"ID":"54452d78-0bbd-4181-afd3-382b44096684","Type":"ContainerDied","Data":"aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832"} Aug 04 13:00:56.493962 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:56.493935 2578 scope.go:117] "RemoveContainer" containerID="aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832" Aug 04 13:00:56.495078 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:56.495053 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:56.495165 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:56.495146 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:56.495205 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:56.495194 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:00:57.500116 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:57.500081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" event={"ID":"54452d78-0bbd-4181-afd3-382b44096684","Type":"ContainerStarted","Data":"6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd"} Aug 04 13:00:57.931623 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:57.931585 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:57.932663 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:57.932635 2578 patch_prober.go:28] interesting pod/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 04 13:00:57.932719 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:57.932704 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 04 13:00:58.419287 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:58.419238 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:00:58.931711 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:58.931676 2578 patch_prober.go:28] interesting pod/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 04 13:00:58.932151 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:58.931735 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 04 13:00:59.931769 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:59.931735 2578 patch_prober.go:28] interesting pod/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 04 13:00:59.932214 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:00:59.931792 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 04 13:01:00.931102 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:00.931061 2578 patch_prober.go:28] interesting pod/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 04 13:01:00.931302 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:00.931170 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 04 13:01:01.932048 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:01.932009 2578 patch_prober.go:28] interesting pod/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Aug 04 13:01:01.932465 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:01.932070 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Aug 04 13:01:02.935544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:02.935515 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:01:02.936541 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:02.936521 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:01:14.577849 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.577803 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh"] Aug 04 13:01:14.578774 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.578741 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" podUID="0ac84152-a75a-4099-bb84-33f309059333" containerName="maas-api" containerID="cri-o://83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04" gracePeriod=30 Aug 04 13:01:14.602415 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.602385 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:14.695590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.695554 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:14.760468 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.760434 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:14.824956 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.824924 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:14.838897 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.838875 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:01:14.926341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.925481 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv9dw\" (UniqueName: \"kubernetes.io/projected/0ac84152-a75a-4099-bb84-33f309059333-kube-api-access-gv9dw\") pod \"0ac84152-a75a-4099-bb84-33f309059333\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " Aug 04 13:01:14.926341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.925591 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls\") pod \"0ac84152-a75a-4099-bb84-33f309059333\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " Aug 04 13:01:14.926341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.925682 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac84152-a75a-4099-bb84-33f309059333-openshift-service-ca\") pod \"0ac84152-a75a-4099-bb84-33f309059333\" (UID: \"0ac84152-a75a-4099-bb84-33f309059333\") " Aug 04 13:01:14.926861 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.926811 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac84152-a75a-4099-bb84-33f309059333-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ac84152-a75a-4099-bb84-33f309059333" (UID: "0ac84152-a75a-4099-bb84-33f309059333"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:01:14.941240 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.929822 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac84152-a75a-4099-bb84-33f309059333-kube-api-access-gv9dw" (OuterVolumeSpecName: "kube-api-access-gv9dw") pod "0ac84152-a75a-4099-bb84-33f309059333" (UID: "0ac84152-a75a-4099-bb84-33f309059333"). InnerVolumeSpecName "kube-api-access-gv9dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:01:14.941240 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.930223 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ac84152-a75a-4099-bb84-33f309059333" (UID: "0ac84152-a75a-4099-bb84-33f309059333"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:01:14.962379 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.954471 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8"] Aug 04 13:01:14.962379 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.954818 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" containerID="cri-o://6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd" gracePeriod=30 Aug 04 13:01:14.962379 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:14.957441 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:15.027107 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.027069 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac84152-a75a-4099-bb84-33f309059333-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:15.027107 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.027101 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac84152-a75a-4099-bb84-33f309059333-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:15.027107 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.027112 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9dw\" (UniqueName: \"kubernetes.io/projected/0ac84152-a75a-4099-bb84-33f309059333-kube-api-access-gv9dw\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:15.183950 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.183835 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:15.600684 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.600653 2578 generic.go:356] "Generic (PLEG): container finished" podID="0ac84152-a75a-4099-bb84-33f309059333" containerID="83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04" exitCode=0 Aug 04 13:01:15.601046 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.600747 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" event={"ID":"0ac84152-a75a-4099-bb84-33f309059333","Type":"ContainerDied","Data":"83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04"} Aug 04 13:01:15.601046 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.600767 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" Aug 04 13:01:15.601046 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.600794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh" event={"ID":"0ac84152-a75a-4099-bb84-33f309059333","Type":"ContainerDied","Data":"ab3e2ccc964eebf7a1f844bb1e459c15807a95f011f3ebb2cd4caf0b7ec6b42e"} Aug 04 13:01:15.601046 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.600822 2578 scope.go:117] "RemoveContainer" containerID="83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04" Aug 04 13:01:15.610162 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.610143 2578 scope.go:117] "RemoveContainer" containerID="83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04" Aug 04 13:01:15.610454 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:01:15.610436 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04\": container with ID starting with 83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04 not found: ID does not exist" containerID="83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04" Aug 04 13:01:15.610513 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.610461 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04"} err="failed to get container status \"83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04\": rpc error: code = NotFound desc = could not find container \"83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04\": container with ID starting with 83c62c23179c4c6a90bb1cbe0716794d026b8e1d575676946371aabeb29dcd04 not found: ID does not exist" Aug 04 13:01:15.632295 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.632259 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh"] Aug 04 13:01:15.646454 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.646430 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cb728584-595dd6986b-q4xbh"] Aug 04 13:01:15.646595 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.646490 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:15.723606 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.723572 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:15.823596 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:15.823510 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:17.554275 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:17.554234 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ac84152-a75a-4099-bb84-33f309059333" path="/var/lib/kubelet/pods/0ac84152-a75a-4099-bb84-33f309059333/volumes" Aug 04 13:01:17.938547 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:17.938508 2578 patch_prober.go:28] interesting pod/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 13:01:17.938740 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:17.938560 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 13:01:20.218058 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.218028 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:01:20.278486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278388 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54452d78-0bbd-4181-afd3-382b44096684-istio-podinfo\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278447 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-envoy\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278485 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54452d78-0bbd-4181-afd3-382b44096684-istiod-ca-cert\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-credential-socket\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278572 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-socket\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278596 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-data\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278619 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-certs\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278687 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm6p8\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-kube-api-access-dm6p8\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.278768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278723 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-istio-token\") pod \"54452d78-0bbd-4181-afd3-382b44096684\" (UID: \"54452d78-0bbd-4181-afd3-382b44096684\") " Aug 04 13:01:20.279043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278863 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:01:20.279043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278929 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-data" (OuterVolumeSpecName: "istio-data") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:01:20.279043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.278980 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54452d78-0bbd-4181-afd3-382b44096684-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:01:20.279043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.279005 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:01:20.279263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.279154 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54452d78-0bbd-4181-afd3-382b44096684-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.279263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.279175 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.279263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.279188 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.279263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.279201 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.279387 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.279287 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:01:20.280859 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.280820 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54452d78-0bbd-4181-afd3-382b44096684-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:01:20.280981 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.280888 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:01:20.280981 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.280948 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-kube-api-access-dm6p8" (OuterVolumeSpecName: "kube-api-access-dm6p8") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "kube-api-access-dm6p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:01:20.281167 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.281151 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-istio-token" (OuterVolumeSpecName: "istio-token") pod "54452d78-0bbd-4181-afd3-382b44096684" (UID: "54452d78-0bbd-4181-afd3-382b44096684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:01:20.380497 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.380461 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm6p8\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-kube-api-access-dm6p8\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.380497 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.380490 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54452d78-0bbd-4181-afd3-382b44096684-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.380497 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.380499 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54452d78-0bbd-4181-afd3-382b44096684-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.380497 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.380506 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.380753 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.380517 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54452d78-0bbd-4181-afd3-382b44096684-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:01:20.630660 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.630628 2578 generic.go:356] "Generic (PLEG): container finished" podID="54452d78-0bbd-4181-afd3-382b44096684" containerID="6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd" exitCode=0 Aug 04 13:01:20.630819 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.630690 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" Aug 04 13:01:20.630819 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.630707 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" event={"ID":"54452d78-0bbd-4181-afd3-382b44096684","Type":"ContainerDied","Data":"6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd"} Aug 04 13:01:20.630819 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.630763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8" event={"ID":"54452d78-0bbd-4181-afd3-382b44096684","Type":"ContainerDied","Data":"ce4e41e76a7a0db731234ec6024dab76635b7b5e1ab83da2a603e0e765961b19"} Aug 04 13:01:20.630819 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.630785 2578 scope.go:117] "RemoveContainer" containerID="6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd" Aug 04 13:01:20.640752 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.640736 2578 scope.go:117] "RemoveContainer" containerID="aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832" Aug 04 13:01:20.649061 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.649046 2578 scope.go:117] "RemoveContainer" containerID="6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd" Aug 04 13:01:20.649344 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:01:20.649320 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd\": container with ID starting with 6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd not found: ID does not exist" containerID="6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd" Aug 04 13:01:20.649445 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.649351 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd"} err="failed to get container status \"6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd\": rpc error: code = NotFound desc = could not find container \"6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd\": container with ID starting with 6c7d8b0918978d786cfcb6a1f0f23058339a7bdcba0005f0a3473097e25847fd not found: ID does not exist" Aug 04 13:01:20.649445 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.649368 2578 scope.go:117] "RemoveContainer" containerID="aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832" Aug 04 13:01:20.649618 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:01:20.649600 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832\": container with ID starting with aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832 not found: ID does not exist" containerID="aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832" Aug 04 13:01:20.649669 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.649625 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832"} err="failed to get container status \"aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832\": rpc error: code = NotFound desc = could not find container \"aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832\": container with ID starting with aa340af25f978f1b6d7259f6092edb544e02687fa730311b0692f96d63918832 not found: ID does not exist" Aug 04 13:01:20.656375 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.656352 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8"] Aug 04 13:01:20.661472 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:20.661450 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cb728584-openshift-default-7c5b49c976-dhfm8"] Aug 04 13:01:21.553537 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:21.553499 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54452d78-0bbd-4181-afd3-382b44096684" path="/var/lib/kubelet/pods/54452d78-0bbd-4181-afd3-382b44096684/volumes" Aug 04 13:01:28.320443 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:28.320402 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:34.256089 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:34.256052 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:34.337750 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:34.337713 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:34.452707 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:34.452673 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:34.552780 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:34.552699 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:34.960668 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:34.960621 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:01:38.463172 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:01:38.463110 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:00.936530 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:00.936494 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:07.555642 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:07.555607 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:07.617093 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:07.617053 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:07.709259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:07.709223 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:07.804790 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:07.804755 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:23.747961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:23.747874 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:30.025062 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:30.025023 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:30.084568 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:30.084532 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:30.164455 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:30.164419 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:30.265890 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:30.265857 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:40.772775 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:40.772739 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:49.585237 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:49.585202 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:02:49.591043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:49.591020 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:02:51.820976 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:51.820941 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:59.103107 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:59.103071 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:59.160676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:59.160640 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:59.236817 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:59.236789 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:59.336475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:59.336444 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:02:59.435262 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:02:59.435174 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:00.483750 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:00.483707 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:00.584169 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:00.584100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:01.820485 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:01.820448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:09.539834 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:09.539796 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:12.531080 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:12.531042 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:18.073244 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.073177 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:18.164172 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.164116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:18.329114 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.329032 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:18.684767 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.684736 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb"] Aug 04 13:03:18.685205 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685188 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" Aug 04 13:03:18.685205 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685205 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" Aug 04 13:03:18.685351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685218 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" Aug 04 13:03:18.685351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685228 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" Aug 04 13:03:18.685351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685246 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" Aug 04 13:03:18.685351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685256 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" Aug 04 13:03:18.685351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685266 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ac84152-a75a-4099-bb84-33f309059333" containerName="maas-api" Aug 04 13:03:18.685351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685275 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ac84152-a75a-4099-bb84-33f309059333" containerName="maas-api" Aug 04 13:03:18.685582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685410 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" Aug 04 13:03:18.685582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685426 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b6358a-9dfe-427c-a46f-959f5c0528bf" containerName="istio-proxy" Aug 04 13:03:18.685582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685438 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" Aug 04 13:03:18.685582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685447 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" Aug 04 13:03:18.685582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.685457 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ac84152-a75a-4099-bb84-33f309059333" containerName="maas-api" Aug 04 13:03:18.688628 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.688614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.692973 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.692953 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d8882e5d-openshift-default-dockercfg-lp647\"" Aug 04 13:03:18.694854 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.694835 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 13:03:18.721617 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.721582 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb"] Aug 04 13:03:18.813662 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-data\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.813840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-socket\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.813840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58177de0-4b44-479c-a402-046abfd6606d-istio-podinfo\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.813840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxsvk\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-kube-api-access-zxsvk\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.813840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-certs\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.813840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-credential-socket\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.814045 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-istio-token\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.814045 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-envoy\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.814045 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.813960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58177de0-4b44-479c-a402-046abfd6606d-istiod-ca-cert\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.914791 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-data\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.914791 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-socket\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58177de0-4b44-479c-a402-046abfd6606d-istio-podinfo\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxsvk\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-kube-api-access-zxsvk\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-certs\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-credential-socket\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-istio-token\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-envoy\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915066 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.914978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58177de0-4b44-479c-a402-046abfd6606d-istiod-ca-cert\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915445 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.915276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-socket\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915445 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.915305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-data\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915445 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.915383 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-certs\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.915454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-credential-socket\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.915764 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.915746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58177de0-4b44-479c-a402-046abfd6606d-istiod-ca-cert\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.917442 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.917419 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-envoy\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.917672 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.917654 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58177de0-4b44-479c-a402-046abfd6606d-istio-podinfo\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.930909 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.930878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-istio-token\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.931544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.931524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxsvk\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-kube-api-access-zxsvk\") pod \"e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:18.999409 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:18.999320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:19.155443 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:19.155412 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb"] Aug 04 13:03:19.157292 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:03:19.157254 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58177de0_4b44_479c_a402_046abfd6606d.slice/crio-cc4817cbe4f91d6f63947605331e0be8a7cbae75d78c531d03a0fc147676e298 WatchSource:0}: Error finding container cc4817cbe4f91d6f63947605331e0be8a7cbae75d78c531d03a0fc147676e298: Status 404 returned error can't find the container with id cc4817cbe4f91d6f63947605331e0be8a7cbae75d78c531d03a0fc147676e298 Aug 04 13:03:19.159401 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:19.159367 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:03:19.159459 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:19.159441 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:03:19.159500 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:19.159470 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:03:19.224100 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:19.224070 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" event={"ID":"58177de0-4b44-479c-a402-046abfd6606d","Type":"ContainerStarted","Data":"cc4817cbe4f91d6f63947605331e0be8a7cbae75d78c531d03a0fc147676e298"} Aug 04 13:03:19.926678 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:19.926642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:20.010720 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:20.010685 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:20.230801 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:20.230699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" event={"ID":"58177de0-4b44-479c-a402-046abfd6606d","Type":"ContainerStarted","Data":"d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df"} Aug 04 13:03:20.258371 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:20.258308 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podStartSLOduration=2.258290642 podStartE2EDuration="2.258290642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:03:20.257711637 +0000 UTC m=+2131.213307456" watchObservedRunningTime="2026-08-04 13:03:20.258290642 +0000 UTC m=+2131.213886435" Aug 04 13:03:21.000145 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:21.000079 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:21.005092 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:21.005065 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:21.236091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:21.236057 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:21.237047 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:21.237026 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:22.332254 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:22.332221 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:22.435417 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:22.435381 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:23.424817 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.424785 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv"] Aug 04 13:03:23.425271 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.425261 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" Aug 04 13:03:23.425343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.425275 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54452d78-0bbd-4181-afd3-382b44096684" containerName="istio-proxy" Aug 04 13:03:23.428415 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.428395 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.436137 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.436107 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d8882e5d-serving-cert\"" Aug 04 13:03:23.449292 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.449260 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv"] Aug 04 13:03:23.493644 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.493604 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:23.565848 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.565812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9hm\" (UniqueName: \"kubernetes.io/projected/8fde05b1-0414-45f3-8dc3-19353afd7544-kube-api-access-dd9hm\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.566018 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.565881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fde05b1-0414-45f3-8dc3-19353afd7544-openshift-service-ca\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.566018 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.565950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fde05b1-0414-45f3-8dc3-19353afd7544-maas-api-tls\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.667115 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.667083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fde05b1-0414-45f3-8dc3-19353afd7544-maas-api-tls\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.667299 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.667191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9hm\" (UniqueName: \"kubernetes.io/projected/8fde05b1-0414-45f3-8dc3-19353afd7544-kube-api-access-dd9hm\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.667299 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.667265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fde05b1-0414-45f3-8dc3-19353afd7544-openshift-service-ca\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.667975 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.667952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fde05b1-0414-45f3-8dc3-19353afd7544-openshift-service-ca\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.669746 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.669716 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fde05b1-0414-45f3-8dc3-19353afd7544-maas-api-tls\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.679378 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.679322 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9hm\" (UniqueName: \"kubernetes.io/projected/8fde05b1-0414-45f3-8dc3-19353afd7544-kube-api-access-dd9hm\") pod \"maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.687833 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.687791 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:23.739639 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.739604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:23.883851 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:23.883827 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv"] Aug 04 13:03:23.885913 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:03:23.885871 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fde05b1_0414_45f3_8dc3_19353afd7544.slice/crio-d231e929308b75b1e9a72873254febf99afe44cde87b88d48b9225810f5f957b WatchSource:0}: Error finding container d231e929308b75b1e9a72873254febf99afe44cde87b88d48b9225810f5f957b: Status 404 returned error can't find the container with id d231e929308b75b1e9a72873254febf99afe44cde87b88d48b9225810f5f957b Aug 04 13:03:24.253697 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.253655 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" event={"ID":"8fde05b1-0414-45f3-8dc3-19353afd7544","Type":"ContainerStarted","Data":"d231e929308b75b1e9a72873254febf99afe44cde87b88d48b9225810f5f957b"} Aug 04 13:03:24.331266 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.331231 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:24.486191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.486151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:24.630820 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.630791 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7"] Aug 04 13:03:24.635535 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.635514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.638658 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.638617 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8882e5d-dockercfg-nb6sk\"" Aug 04 13:03:24.638902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.638886 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8882e5d\"" Aug 04 13:03:24.650589 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.650565 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7"] Aug 04 13:03:24.778194 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.778164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpvr\" (UniqueName: \"kubernetes.io/projected/b89ef06c-021d-4a65-b132-d2f51216aa21-kube-api-access-bqpvr\") pod \"payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.778319 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.778265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89ef06c-021d-4a65-b132-d2f51216aa21-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.879671 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.879636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpvr\" (UniqueName: \"kubernetes.io/projected/b89ef06c-021d-4a65-b132-d2f51216aa21-kube-api-access-bqpvr\") pod \"payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.879861 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.879756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89ef06c-021d-4a65-b132-d2f51216aa21-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.880468 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.880444 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89ef06c-021d-4a65-b132-d2f51216aa21-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.894174 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.894082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpvr\" (UniqueName: \"kubernetes.io/projected/b89ef06c-021d-4a65-b132-d2f51216aa21-kube-api-access-bqpvr\") pod \"payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:24.949152 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:24.949100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:25.098597 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.098558 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7"] Aug 04 13:03:25.100742 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:03:25.100707 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89ef06c_021d_4a65_b132_d2f51216aa21.slice/crio-32c7dba0890d437a26286021d8cd091b0496d86eef970048a5c1d3183599b0ee WatchSource:0}: Error finding container 32c7dba0890d437a26286021d8cd091b0496d86eef970048a5c1d3183599b0ee: Status 404 returned error can't find the container with id 32c7dba0890d437a26286021d8cd091b0496d86eef970048a5c1d3183599b0ee Aug 04 13:03:25.260624 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.260584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" event={"ID":"b89ef06c-021d-4a65-b132-d2f51216aa21","Type":"ContainerStarted","Data":"4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046"} Aug 04 13:03:25.260817 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.260650 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:25.260817 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.260667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" event={"ID":"b89ef06c-021d-4a65-b132-d2f51216aa21","Type":"ContainerStarted","Data":"32c7dba0890d437a26286021d8cd091b0496d86eef970048a5c1d3183599b0ee"} Aug 04 13:03:25.262185 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.262159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" event={"ID":"8fde05b1-0414-45f3-8dc3-19353afd7544","Type":"ContainerStarted","Data":"85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de"} Aug 04 13:03:25.262306 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.262209 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:25.283465 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.283415 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" podStartSLOduration=1.283398163 podStartE2EDuration="1.283398163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:03:25.282293162 +0000 UTC m=+2136.237888961" watchObservedRunningTime="2026-08-04 13:03:25.283398163 +0000 UTC m=+2136.238993956" Aug 04 13:03:25.308912 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:25.308857 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" podStartSLOduration=1.459268513 podStartE2EDuration="2.308838881s" totalImagesPullingTime="849.570368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:03:25.306898832 +0000 UTC m=+2136.262494622" watchObservedRunningTime="2026-08-04 13:03:25.308838881 +0000 UTC m=+2136.264434675" Aug 04 13:03:31.273824 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:31.273790 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:03:32.093408 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:32.093374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:36.270544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:36.270512 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:03:36.893851 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:36.893817 2578 patch_prober.go:28] interesting pod/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:39774->10.134.0.57:15021: read: connection reset by peer" start-of-body= Aug 04 13:03:36.894010 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:36.893880 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:39774->10.134.0.57:15021: read: connection reset by peer" Aug 04 13:03:37.328388 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:37.328302 2578 generic.go:356] "Generic (PLEG): container finished" podID="58177de0-4b44-479c-a402-046abfd6606d" containerID="d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df" exitCode=0 Aug 04 13:03:37.328388 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:37.328377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" event={"ID":"58177de0-4b44-479c-a402-046abfd6606d","Type":"ContainerDied","Data":"d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df"} Aug 04 13:03:37.328807 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:37.328792 2578 scope.go:117] "RemoveContainer" containerID="d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df" Aug 04 13:03:37.329937 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:37.329910 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:03:37.330020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:37.329990 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:03:37.330063 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:37.330020 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:03:38.335577 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:38.335540 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" event={"ID":"58177de0-4b44-479c-a402-046abfd6606d","Type":"ContainerStarted","Data":"590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7"} Aug 04 13:03:39.000451 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:39.000408 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:39.000712 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:39.000686 2578 patch_prober.go:28] interesting pod/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 04 13:03:39.000800 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:39.000750 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 04 13:03:40.000592 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:40.000506 2578 patch_prober.go:28] interesting pod/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 04 13:03:40.000592 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:40.000570 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 04 13:03:41.000146 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:41.000087 2578 patch_prober.go:28] interesting pod/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 04 13:03:41.000336 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:41.000189 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 04 13:03:41.999930 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:41.999892 2578 patch_prober.go:28] interesting pod/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 04 13:03:42.000354 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:41.999969 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 04 13:03:43.000359 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:43.000320 2578 patch_prober.go:28] interesting pod/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Aug 04 13:03:43.000744 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:43.000389 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Aug 04 13:03:44.003361 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:44.003327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:44.003741 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:44.003713 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:44.004574 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:44.004556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:03:52.404400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:52.404364 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:53.233366 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:53.233311 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:53.334923 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:53.334886 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:03:53.539674 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:03:53.539588 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:00.214964 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:00.214931 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:00.277544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:00.277489 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:11.434425 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.434385 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv"] Aug 04 13:04:11.434940 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.434604 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" podUID="8fde05b1-0414-45f3-8dc3-19353afd7544" containerName="maas-api" containerID="cri-o://85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de" gracePeriod=30 Aug 04 13:04:11.464649 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.464616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:11.555915 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.555887 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7"] Aug 04 13:04:11.556205 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.556105 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" podUID="b89ef06c-021d-4a65-b132-d2f51216aa21" containerName="payload-processing" containerID="cri-o://4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046" gracePeriod=30 Aug 04 13:04:11.597948 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.597869 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:11.697671 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.697637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:11.705979 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.705946 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:04:11.800754 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.800718 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:11.817008 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.816983 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:04:11.857322 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.857284 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fde05b1-0414-45f3-8dc3-19353afd7544-openshift-service-ca\") pod \"8fde05b1-0414-45f3-8dc3-19353afd7544\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " Aug 04 13:04:11.857517 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.857380 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd9hm\" (UniqueName: \"kubernetes.io/projected/8fde05b1-0414-45f3-8dc3-19353afd7544-kube-api-access-dd9hm\") pod \"8fde05b1-0414-45f3-8dc3-19353afd7544\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " Aug 04 13:04:11.857517 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.857413 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fde05b1-0414-45f3-8dc3-19353afd7544-maas-api-tls\") pod \"8fde05b1-0414-45f3-8dc3-19353afd7544\" (UID: \"8fde05b1-0414-45f3-8dc3-19353afd7544\") " Aug 04 13:04:11.857877 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.857730 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fde05b1-0414-45f3-8dc3-19353afd7544-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fde05b1-0414-45f3-8dc3-19353afd7544" (UID: "8fde05b1-0414-45f3-8dc3-19353afd7544"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:04:11.857930 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.857913 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fde05b1-0414-45f3-8dc3-19353afd7544-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:11.859854 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.859823 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fde05b1-0414-45f3-8dc3-19353afd7544-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fde05b1-0414-45f3-8dc3-19353afd7544" (UID: "8fde05b1-0414-45f3-8dc3-19353afd7544"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:04:11.860206 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.860186 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fde05b1-0414-45f3-8dc3-19353afd7544-kube-api-access-dd9hm" (OuterVolumeSpecName: "kube-api-access-dd9hm") pod "8fde05b1-0414-45f3-8dc3-19353afd7544" (UID: "8fde05b1-0414-45f3-8dc3-19353afd7544"). InnerVolumeSpecName "kube-api-access-dd9hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:04:11.959035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.958941 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89ef06c-021d-4a65-b132-d2f51216aa21-plugins-config-volume\") pod \"b89ef06c-021d-4a65-b132-d2f51216aa21\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " Aug 04 13:04:11.959232 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.959091 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqpvr\" (UniqueName: \"kubernetes.io/projected/b89ef06c-021d-4a65-b132-d2f51216aa21-kube-api-access-bqpvr\") pod \"b89ef06c-021d-4a65-b132-d2f51216aa21\" (UID: \"b89ef06c-021d-4a65-b132-d2f51216aa21\") " Aug 04 13:04:11.959303 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.959290 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9hm\" (UniqueName: \"kubernetes.io/projected/8fde05b1-0414-45f3-8dc3-19353afd7544-kube-api-access-dd9hm\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:11.959347 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.959305 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fde05b1-0414-45f3-8dc3-19353afd7544-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:11.959448 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.959422 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89ef06c-021d-4a65-b132-d2f51216aa21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b89ef06c-021d-4a65-b132-d2f51216aa21" (UID: "b89ef06c-021d-4a65-b132-d2f51216aa21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:04:11.961287 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:11.961266 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89ef06c-021d-4a65-b132-d2f51216aa21-kube-api-access-bqpvr" (OuterVolumeSpecName: "kube-api-access-bqpvr") pod "b89ef06c-021d-4a65-b132-d2f51216aa21" (UID: "b89ef06c-021d-4a65-b132-d2f51216aa21"). InnerVolumeSpecName "kube-api-access-bqpvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:04:12.059994 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.059953 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqpvr\" (UniqueName: \"kubernetes.io/projected/b89ef06c-021d-4a65-b132-d2f51216aa21-kube-api-access-bqpvr\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:12.059994 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.059986 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89ef06c-021d-4a65-b132-d2f51216aa21-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:12.509740 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.509702 2578 generic.go:356] "Generic (PLEG): container finished" podID="b89ef06c-021d-4a65-b132-d2f51216aa21" containerID="4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046" exitCode=0 Aug 04 13:04:12.510228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.509763 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" Aug 04 13:04:12.510228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.509793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" event={"ID":"b89ef06c-021d-4a65-b132-d2f51216aa21","Type":"ContainerDied","Data":"4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046"} Aug 04 13:04:12.510228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.509885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7" event={"ID":"b89ef06c-021d-4a65-b132-d2f51216aa21","Type":"ContainerDied","Data":"32c7dba0890d437a26286021d8cd091b0496d86eef970048a5c1d3183599b0ee"} Aug 04 13:04:12.510228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.509919 2578 scope.go:117] "RemoveContainer" containerID="4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046" Aug 04 13:04:12.511233 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.511115 2578 generic.go:356] "Generic (PLEG): container finished" podID="8fde05b1-0414-45f3-8dc3-19353afd7544" containerID="85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de" exitCode=0 Aug 04 13:04:12.511233 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.511193 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" Aug 04 13:04:12.511233 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.511215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" event={"ID":"8fde05b1-0414-45f3-8dc3-19353afd7544","Type":"ContainerDied","Data":"85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de"} Aug 04 13:04:12.511419 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.511258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv" event={"ID":"8fde05b1-0414-45f3-8dc3-19353afd7544","Type":"ContainerDied","Data":"d231e929308b75b1e9a72873254febf99afe44cde87b88d48b9225810f5f957b"} Aug 04 13:04:12.520493 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.520472 2578 scope.go:117] "RemoveContainer" containerID="4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046" Aug 04 13:04:12.520802 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:04:12.520777 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046\": container with ID starting with 4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046 not found: ID does not exist" containerID="4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046" Aug 04 13:04:12.520873 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.520810 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046"} err="failed to get container status \"4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046\": rpc error: code = NotFound desc = could not find container \"4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046\": container with ID starting with 4369fd321fa4d283e300c6cc2ee8e664fa3d1fb8242605b8624ee4c0d1192046 not found: ID does not exist" Aug 04 13:04:12.520873 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.520835 2578 scope.go:117] "RemoveContainer" containerID="85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de" Aug 04 13:04:12.529462 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.529442 2578 scope.go:117] "RemoveContainer" containerID="85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de" Aug 04 13:04:12.529744 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:04:12.529723 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de\": container with ID starting with 85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de not found: ID does not exist" containerID="85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de" Aug 04 13:04:12.529819 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.529752 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de"} err="failed to get container status \"85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de\": rpc error: code = NotFound desc = could not find container \"85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de\": container with ID starting with 85185db2354cd68b791c25c591f95d211e5e7d3304654de4f6d87f6fc62376de not found: ID does not exist" Aug 04 13:04:12.540177 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.540153 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7"] Aug 04 13:04:12.545513 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.545489 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8882e5d-5767fd685d-6l2c7"] Aug 04 13:04:12.564495 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.564471 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv"] Aug 04 13:04:12.566652 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.566629 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8882e5d-cb58cff8c-rbfxv"] Aug 04 13:04:12.975527 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:12.975493 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:13.553438 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:13.553404 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fde05b1-0414-45f3-8dc3-19353afd7544" path="/var/lib/kubelet/pods/8fde05b1-0414-45f3-8dc3-19353afd7544/volumes" Aug 04 13:04:13.553813 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:13.553761 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89ef06c-021d-4a65-b132-d2f51216aa21" path="/var/lib/kubelet/pods/b89ef06c-021d-4a65-b132-d2f51216aa21/volumes" Aug 04 13:04:18.975416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:18.975380 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:19.077514 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.077472 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:19.237991 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.237905 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb"] Aug 04 13:04:19.238186 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.238139 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" containerID="cri-o://590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7" gracePeriod=30 Aug 04 13:04:19.265658 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.265622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:19.518095 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.518009 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:19.618456 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.618426 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:19.718314 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.718282 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:19.864529 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:19.864493 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:24.486435 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.486403 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:04:24.574013 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.573922 2578 generic.go:356] "Generic (PLEG): container finished" podID="58177de0-4b44-479c-a402-046abfd6606d" containerID="590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7" exitCode=0 Aug 04 13:04:24.574013 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.573970 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" Aug 04 13:04:24.574236 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.574008 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" event={"ID":"58177de0-4b44-479c-a402-046abfd6606d","Type":"ContainerDied","Data":"590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7"} Aug 04 13:04:24.574236 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.574069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb" event={"ID":"58177de0-4b44-479c-a402-046abfd6606d","Type":"ContainerDied","Data":"cc4817cbe4f91d6f63947605331e0be8a7cbae75d78c531d03a0fc147676e298"} Aug 04 13:04:24.574236 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.574098 2578 scope.go:117] "RemoveContainer" containerID="590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7" Aug 04 13:04:24.576335 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576311 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-envoy\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576452 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxsvk\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-kube-api-access-zxsvk\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576452 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-certs\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576452 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576408 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-socket\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576452 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-istio-token\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576665 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576486 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-data\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576665 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576520 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58177de0-4b44-479c-a402-046abfd6606d-istiod-ca-cert\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576665 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576571 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-credential-socket\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576665 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576602 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58177de0-4b44-479c-a402-046abfd6606d-istio-podinfo\") pod \"58177de0-4b44-479c-a402-046abfd6606d\" (UID: \"58177de0-4b44-479c-a402-046abfd6606d\") " Aug 04 13:04:24.576872 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576802 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:04:24.576928 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576893 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-data" (OuterVolumeSpecName: "istio-data") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:04:24.577007 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576975 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58177de0-4b44-479c-a402-046abfd6606d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:04:24.577007 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.576996 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.577138 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.577014 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.577138 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.577104 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:04:24.577329 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.577304 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:04:24.579047 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.579023 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58177de0-4b44-479c-a402-046abfd6606d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:04:24.579280 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.579254 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-kube-api-access-zxsvk" (OuterVolumeSpecName: "kube-api-access-zxsvk") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "kube-api-access-zxsvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:04:24.579415 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.579319 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:04:24.579415 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.579378 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-istio-token" (OuterVolumeSpecName: "istio-token") pod "58177de0-4b44-479c-a402-046abfd6606d" (UID: "58177de0-4b44-479c-a402-046abfd6606d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:04:24.583584 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.583567 2578 scope.go:117] "RemoveContainer" containerID="d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df" Aug 04 13:04:24.593294 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.593266 2578 scope.go:117] "RemoveContainer" containerID="590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7" Aug 04 13:04:24.593571 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:04:24.593551 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7\": container with ID starting with 590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7 not found: ID does not exist" containerID="590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7" Aug 04 13:04:24.593648 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.593584 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7"} err="failed to get container status \"590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7\": rpc error: code = NotFound desc = could not find container \"590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7\": container with ID starting with 590f0528245e06342bf3f799581a7ca2a8f82ded845098ce7fba15e09a32ccf7 not found: ID does not exist" Aug 04 13:04:24.593648 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.593612 2578 scope.go:117] "RemoveContainer" containerID="d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df" Aug 04 13:04:24.593890 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:04:24.593873 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df\": container with ID starting with d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df not found: ID does not exist" containerID="d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df" Aug 04 13:04:24.593931 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.593896 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df"} err="failed to get container status \"d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df\": rpc error: code = NotFound desc = could not find container \"d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df\": container with ID starting with d520f1c972c633c561abbe706204d1f0fe9490959fb9cd5a2ead9ff7b84fc4df not found: ID does not exist" Aug 04 13:04:24.677807 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677765 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.677807 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677799 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxsvk\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-kube-api-access-zxsvk\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.677807 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677810 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.677807 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677818 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58177de0-4b44-479c-a402-046abfd6606d-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.678081 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677827 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58177de0-4b44-479c-a402-046abfd6606d-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.678081 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677834 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58177de0-4b44-479c-a402-046abfd6606d-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.678081 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.677842 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58177de0-4b44-479c-a402-046abfd6606d-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:04:24.900494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.900460 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb"] Aug 04 13:04:24.905661 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:24.905632 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d8882e5d-openshift-default-54b94949dd-md5zb"] Aug 04 13:04:25.552926 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:25.552889 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58177de0-4b44-479c-a402-046abfd6606d" path="/var/lib/kubelet/pods/58177de0-4b44-479c-a402-046abfd6606d/volumes" Aug 04 13:04:26.684393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:26.684357 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:26.758698 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:26.758661 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:26.879750 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:26.879720 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:27.664455 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:27.664403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:27.762530 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:27.762498 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:33.804361 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:33.804326 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:04:33.901242 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:04:33.901208 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:05:08.348440 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:05:08.348409 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:01.173584 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:01.173546 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:36.376733 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.376697 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:36.672067 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.671976 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28"] Aug 04 13:06:36.672401 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672383 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" Aug 04 13:06:36.672401 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672398 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672417 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672422 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672428 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89ef06c-021d-4a65-b132-d2f51216aa21" containerName="payload-processing" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672434 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89ef06c-021d-4a65-b132-d2f51216aa21" containerName="payload-processing" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672444 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fde05b1-0414-45f3-8dc3-19353afd7544" containerName="maas-api" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672449 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fde05b1-0414-45f3-8dc3-19353afd7544" containerName="maas-api" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672519 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fde05b1-0414-45f3-8dc3-19353afd7544" containerName="maas-api" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672527 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89ef06c-021d-4a65-b132-d2f51216aa21" containerName="payload-processing" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672533 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" Aug 04 13:06:36.672576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.672539 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58177de0-4b44-479c-a402-046abfd6606d" containerName="istio-proxy" Aug 04 13:06:36.675750 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.675735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.678820 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.678796 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f758d543-openshift-default-dockercfg-796cj\"" Aug 04 13:06:36.678951 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.678864 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 13:06:36.688615 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.688591 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28"] Aug 04 13:06:36.708590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.708537 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:36.811568 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.811534 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:36.829076 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sl9p\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-kube-api-access-6sl9p\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829230 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-podinfo\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829230 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-certs\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829230 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-token\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829230 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-data\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829390 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-socket\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829390 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-credential-socket\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829390 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-envoy\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.829486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.829420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2af373-af3a-49d3-a022-9749ac02ce19-istiod-ca-cert\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930409 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-socket\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930409 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-credential-socket\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-envoy\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2af373-af3a-49d3-a022-9749ac02ce19-istiod-ca-cert\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl9p\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-kube-api-access-6sl9p\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-podinfo\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-certs\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-token\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-data\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930979 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-socket\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.930979 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-data\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.931068 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.930987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-certs\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.931237 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.931216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-credential-socket\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.931308 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.931238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2af373-af3a-49d3-a022-9749ac02ce19-istiod-ca-cert\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.933458 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.933436 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-envoy\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.933616 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.933601 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-podinfo\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.942552 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.942523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-token\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.942816 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.942795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl9p\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-kube-api-access-6sl9p\") pod \"e2e-mt-f758d543-openshift-default-545475f656-lnp28\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:36.987230 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:36.987200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:37.131794 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.131760 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28"] Aug 04 13:06:37.133944 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:06:37.133915 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2af373_af3a_49d3_a022_9749ac02ce19.slice/crio-779aba562a3c77a861ba2978433c009f65193d2f578bf6ef4dd58e048181c345 WatchSource:0}: Error finding container 779aba562a3c77a861ba2978433c009f65193d2f578bf6ef4dd58e048181c345: Status 404 returned error can't find the container with id 779aba562a3c77a861ba2978433c009f65193d2f578bf6ef4dd58e048181c345 Aug 04 13:06:37.135964 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.135933 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:06:37.136046 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.135998 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:06:37.136046 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.136025 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:06:37.226972 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.226939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" event={"ID":"9c2af373-af3a-49d3-a022-9749ac02ce19","Type":"ContainerStarted","Data":"94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f"} Aug 04 13:06:37.227157 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.226991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" event={"ID":"9c2af373-af3a-49d3-a022-9749ac02ce19","Type":"ContainerStarted","Data":"779aba562a3c77a861ba2978433c009f65193d2f578bf6ef4dd58e048181c345"} Aug 04 13:06:37.255236 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.255178 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podStartSLOduration=1.255161485 podStartE2EDuration="1.255161485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:06:37.253793072 +0000 UTC m=+2328.209388922" watchObservedRunningTime="2026-08-04 13:06:37.255161485 +0000 UTC m=+2328.210757279" Aug 04 13:06:37.988018 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:37.987976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:38.993306 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:38.993276 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:39.212323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:39.212275 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:39.237586 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:39.237554 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:39.238650 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:39.238632 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:39.317714 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:39.317620 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:40.354726 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.354692 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz"] Aug 04 13:06:40.359041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.359015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.364039 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.364014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f758d543-serving-cert\"" Aug 04 13:06:40.374270 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.374240 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz"] Aug 04 13:06:40.395966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.395921 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:40.463667 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.463627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5wg\" (UniqueName: \"kubernetes.io/projected/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-kube-api-access-4m5wg\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.463852 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.463692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-openshift-service-ca\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.463852 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.463723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-maas-api-tls\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.496286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.496237 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:40.564742 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.564703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-openshift-service-ca\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.564742 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.564748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-maas-api-tls\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.565001 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.564847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5wg\" (UniqueName: \"kubernetes.io/projected/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-kube-api-access-4m5wg\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.565378 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.565357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-openshift-service-ca\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.567368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.567341 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-maas-api-tls\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.575641 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.575612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5wg\" (UniqueName: \"kubernetes.io/projected/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-kube-api-access-4m5wg\") pod \"maas-api-e2e-mt-f758d543-7844986d8c-chqhz\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.673962 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.673873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:40.822262 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.822230 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz"] Aug 04 13:06:40.825524 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:06:40.825476 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc7e423_beff_46de_ba6c_8ef3cf441ab5.slice/crio-d2cb6d195d9016892692c4505799cb18c581187781024b75a275547cc52f2b51 WatchSource:0}: Error finding container d2cb6d195d9016892692c4505799cb18c581187781024b75a275547cc52f2b51: Status 404 returned error can't find the container with id d2cb6d195d9016892692c4505799cb18c581187781024b75a275547cc52f2b51 Aug 04 13:06:40.827023 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:40.827000 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 13:06:41.025529 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.025431 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:41.128502 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.128463 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:06:41.166936 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.166899 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4"] Aug 04 13:06:41.171680 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.171662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.174775 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.174752 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f758d543\"" Aug 04 13:06:41.174906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.174863 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f758d543-dockercfg-j7jbl\"" Aug 04 13:06:41.182169 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.182139 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4"] Aug 04 13:06:41.248733 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.248702 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" event={"ID":"fcc7e423-beff-46de-ba6c-8ef3cf441ab5","Type":"ContainerStarted","Data":"d2cb6d195d9016892692c4505799cb18c581187781024b75a275547cc52f2b51"} Aug 04 13:06:41.277667 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.277579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb6r\" (UniqueName: \"kubernetes.io/projected/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-kube-api-access-8mb6r\") pod \"payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.277667 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.277636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-plugins-config-volume\") pod \"payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.378808 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.378765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb6r\" (UniqueName: \"kubernetes.io/projected/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-kube-api-access-8mb6r\") pod \"payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.379278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.378856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-plugins-config-volume\") pod \"payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.379698 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.379677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-plugins-config-volume\") pod \"payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.392369 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.392342 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb6r\" (UniqueName: \"kubernetes.io/projected/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-kube-api-access-8mb6r\") pod \"payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.484523 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.484485 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:41.633399 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:41.633365 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4"] Aug 04 13:06:41.636431 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:06:41.636394 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d393b8f_12c3_419c_a5dc_3b3c287dafbf.slice/crio-581aa73c7d5f69711374b5547c59f1b0dfb5bb670df543d3318494df79c185f5 WatchSource:0}: Error finding container 581aa73c7d5f69711374b5547c59f1b0dfb5bb670df543d3318494df79c185f5: Status 404 returned error can't find the container with id 581aa73c7d5f69711374b5547c59f1b0dfb5bb670df543d3318494df79c185f5 Aug 04 13:06:42.256735 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.256632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" event={"ID":"fcc7e423-beff-46de-ba6c-8ef3cf441ab5","Type":"ContainerStarted","Data":"f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d"} Aug 04 13:06:42.256911 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.256757 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:42.258052 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.258021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" event={"ID":"1d393b8f-12c3-419c-a5dc-3b3c287dafbf","Type":"ContainerStarted","Data":"0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c"} Aug 04 13:06:42.258204 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.258079 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:42.258204 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.258099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" event={"ID":"1d393b8f-12c3-419c-a5dc-3b3c287dafbf","Type":"ContainerStarted","Data":"581aa73c7d5f69711374b5547c59f1b0dfb5bb670df543d3318494df79c185f5"} Aug 04 13:06:42.293464 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.293405 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" podStartSLOduration=1.164747994 podStartE2EDuration="2.293383888s" totalImagesPullingTime="1.128635894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:06:42.274955496 +0000 UTC m=+2333.230551302" watchObservedRunningTime="2026-08-04 13:06:42.293383888 +0000 UTC m=+2333.248979682" Aug 04 13:06:42.294398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:42.294358 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" podStartSLOduration=1.294343867 podStartE2EDuration="1.294343867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:06:42.29362702 +0000 UTC m=+2333.249222826" watchObservedRunningTime="2026-08-04 13:06:42.294343867 +0000 UTC m=+2333.249939687" Aug 04 13:06:48.268769 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:48.268741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:06:53.266176 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.266058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:06:53.314747 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.314702 2578 generic.go:356] "Generic (PLEG): container finished" podID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerID="94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f" exitCode=0 Aug 04 13:06:53.314939 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.314778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" event={"ID":"9c2af373-af3a-49d3-a022-9749ac02ce19","Type":"ContainerDied","Data":"94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f"} Aug 04 13:06:53.315308 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.315288 2578 scope.go:117] "RemoveContainer" containerID="94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f" Aug 04 13:06:53.316510 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.316484 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:06:53.316620 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.316562 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:06:53.316620 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:53.316614 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:06:54.237803 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:54.237745 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:54.325656 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:54.325616 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" event={"ID":"9c2af373-af3a-49d3-a022-9749ac02ce19","Type":"ContainerStarted","Data":"f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd"} Aug 04 13:06:54.987635 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:54.987598 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:54.988843 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:54.988812 2578 patch_prober.go:28] interesting pod/e2e-mt-f758d543-openshift-default-545475f656-lnp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Aug 04 13:06:54.988979 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:54.988881 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Aug 04 13:06:55.987934 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:55.987899 2578 patch_prober.go:28] interesting pod/e2e-mt-f758d543-openshift-default-545475f656-lnp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Aug 04 13:06:55.988358 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:55.987958 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Aug 04 13:06:56.988490 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:56.988458 2578 patch_prober.go:28] interesting pod/e2e-mt-f758d543-openshift-default-545475f656-lnp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Aug 04 13:06:56.988860 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:56.988513 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Aug 04 13:06:57.987992 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:57.987959 2578 patch_prober.go:28] interesting pod/e2e-mt-f758d543-openshift-default-545475f656-lnp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Aug 04 13:06:57.988188 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:57.988013 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Aug 04 13:06:58.988546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:58.988513 2578 patch_prober.go:28] interesting pod/e2e-mt-f758d543-openshift-default-545475f656-lnp28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Aug 04 13:06:58.988933 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:58.988567 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Aug 04 13:06:59.991855 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:59.991821 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:59.992366 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:59.992346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:06:59.992984 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:06:59.992967 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:07:09.343371 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:07:09.343337 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:07:25.592642 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:07:25.592602 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:07:38.468397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:07:38.468359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:07:38.569804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:07:38.569771 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:07:49.618798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:07:49.618753 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:07:49.625920 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:07:49.625897 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:08:10.040813 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.040718 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz"] Aug 04 13:08:10.041297 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.040982 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" podUID="fcc7e423-beff-46de-ba6c-8ef3cf441ab5" containerName="maas-api" containerID="cri-o://f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d" gracePeriod=30 Aug 04 13:08:10.056621 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.056583 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:10.089855 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.089818 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4"] Aug 04 13:08:10.090044 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.090017 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" podUID="1d393b8f-12c3-419c-a5dc-3b3c287dafbf" containerName="payload-processing" containerID="cri-o://0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c" gracePeriod=30 Aug 04 13:08:10.156159 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.156110 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:10.254300 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.254265 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:10.315483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.315453 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:10.340995 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.340968 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:08:10.368805 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.368778 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-maas-api-tls\") pod \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " Aug 04 13:08:10.368970 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.368826 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m5wg\" (UniqueName: \"kubernetes.io/projected/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-kube-api-access-4m5wg\") pod \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " Aug 04 13:08:10.369033 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.368998 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-openshift-service-ca\") pod \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\" (UID: \"fcc7e423-beff-46de-ba6c-8ef3cf441ab5\") " Aug 04 13:08:10.369453 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.369388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcc7e423-beff-46de-ba6c-8ef3cf441ab5" (UID: "fcc7e423-beff-46de-ba6c-8ef3cf441ab5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:08:10.371509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.371475 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcc7e423-beff-46de-ba6c-8ef3cf441ab5" (UID: "fcc7e423-beff-46de-ba6c-8ef3cf441ab5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:08:10.371629 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.371562 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-kube-api-access-4m5wg" (OuterVolumeSpecName: "kube-api-access-4m5wg") pod "fcc7e423-beff-46de-ba6c-8ef3cf441ab5" (UID: "fcc7e423-beff-46de-ba6c-8ef3cf441ab5"). InnerVolumeSpecName "kube-api-access-4m5wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:08:10.374605 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.374581 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:08:10.470509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.470455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-plugins-config-volume\") pod \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " Aug 04 13:08:10.470509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.470524 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mb6r\" (UniqueName: \"kubernetes.io/projected/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-kube-api-access-8mb6r\") pod \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\" (UID: \"1d393b8f-12c3-419c-a5dc-3b3c287dafbf\") " Aug 04 13:08:10.470745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.470708 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:10.470745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.470719 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:10.470745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.470727 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5wg\" (UniqueName: \"kubernetes.io/projected/fcc7e423-beff-46de-ba6c-8ef3cf441ab5-kube-api-access-4m5wg\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:10.470874 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.470848 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d393b8f-12c3-419c-a5dc-3b3c287dafbf" (UID: "1d393b8f-12c3-419c-a5dc-3b3c287dafbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:08:10.472840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.472812 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-kube-api-access-8mb6r" (OuterVolumeSpecName: "kube-api-access-8mb6r") pod "1d393b8f-12c3-419c-a5dc-3b3c287dafbf" (UID: "1d393b8f-12c3-419c-a5dc-3b3c287dafbf"). InnerVolumeSpecName "kube-api-access-8mb6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:08:10.571673 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.571565 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:10.571673 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.571615 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb6r\" (UniqueName: \"kubernetes.io/projected/1d393b8f-12c3-419c-a5dc-3b3c287dafbf-kube-api-access-8mb6r\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:10.712333 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.712296 2578 generic.go:356] "Generic (PLEG): container finished" podID="fcc7e423-beff-46de-ba6c-8ef3cf441ab5" containerID="f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d" exitCode=0 Aug 04 13:08:10.712509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.712347 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" Aug 04 13:08:10.712509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.712387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" event={"ID":"fcc7e423-beff-46de-ba6c-8ef3cf441ab5","Type":"ContainerDied","Data":"f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d"} Aug 04 13:08:10.712509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.712454 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz" event={"ID":"fcc7e423-beff-46de-ba6c-8ef3cf441ab5","Type":"ContainerDied","Data":"d2cb6d195d9016892692c4505799cb18c581187781024b75a275547cc52f2b51"} Aug 04 13:08:10.712509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.712483 2578 scope.go:117] "RemoveContainer" containerID="f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d" Aug 04 13:08:10.713799 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.713766 2578 generic.go:356] "Generic (PLEG): container finished" podID="1d393b8f-12c3-419c-a5dc-3b3c287dafbf" containerID="0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c" exitCode=0 Aug 04 13:08:10.713899 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.713804 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" Aug 04 13:08:10.713899 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.713860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" event={"ID":"1d393b8f-12c3-419c-a5dc-3b3c287dafbf","Type":"ContainerDied","Data":"0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c"} Aug 04 13:08:10.714001 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.713918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4" event={"ID":"1d393b8f-12c3-419c-a5dc-3b3c287dafbf","Type":"ContainerDied","Data":"581aa73c7d5f69711374b5547c59f1b0dfb5bb670df543d3318494df79c185f5"} Aug 04 13:08:10.721782 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.721762 2578 scope.go:117] "RemoveContainer" containerID="f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d" Aug 04 13:08:10.722145 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:08:10.722097 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d\": container with ID starting with f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d not found: ID does not exist" containerID="f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d" Aug 04 13:08:10.722216 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.722157 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d"} err="failed to get container status \"f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d\": rpc error: code = NotFound desc = could not find container \"f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d\": container with ID starting with f865e7e500ae3904fbbc49e7d4b51605d84b66f3e650cc673242e78d3193612d not found: ID does not exist" Aug 04 13:08:10.722216 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.722175 2578 scope.go:117] "RemoveContainer" containerID="0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c" Aug 04 13:08:10.730365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.730344 2578 scope.go:117] "RemoveContainer" containerID="0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c" Aug 04 13:08:10.730625 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:08:10.730601 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c\": container with ID starting with 0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c not found: ID does not exist" containerID="0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c" Aug 04 13:08:10.730684 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.730636 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c"} err="failed to get container status \"0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c\": rpc error: code = NotFound desc = could not find container \"0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c\": container with ID starting with 0a6d6192d1cb337a89b6e82c9d25a117387b73a5b910eaf54f04349d3c96889c not found: ID does not exist" Aug 04 13:08:10.740588 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.740560 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz"] Aug 04 13:08:10.745131 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.745087 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f758d543-7844986d8c-chqhz"] Aug 04 13:08:10.758222 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.758184 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4"] Aug 04 13:08:10.763080 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:10.763051 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f758d543-d6978cdfd-5v9g4"] Aug 04 13:08:11.553538 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:11.553510 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d393b8f-12c3-419c-a5dc-3b3c287dafbf" path="/var/lib/kubelet/pods/1d393b8f-12c3-419c-a5dc-3b3c287dafbf/volumes" Aug 04 13:08:11.553904 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:11.553883 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc7e423-beff-46de-ba6c-8ef3cf441ab5" path="/var/lib/kubelet/pods/fcc7e423-beff-46de-ba6c-8ef3cf441ab5/volumes" Aug 04 13:08:13.595560 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:13.595514 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:18.672973 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:18.672932 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:34.435172 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:34.435111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:35.290301 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:35.290266 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:36.426444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:36.426407 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:43.030313 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:43.030277 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28"] Aug 04 13:08:43.030856 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:43.030503 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" containerID="cri-o://f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd" gracePeriod=30 Aug 04 13:08:44.352849 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:44.352813 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:44.456293 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:44.456260 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:44.994732 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:44.994687 2578 patch_prober.go:28] interesting pod/e2e-mt-f758d543-openshift-default-545475f656-lnp28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 13:08:44.994911 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:44.994749 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 13:08:48.286910 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.286884 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:08:48.343187 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2af373-af3a-49d3-a022-9749ac02ce19-istiod-ca-cert\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343363 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343256 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-data\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343363 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343301 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sl9p\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-kube-api-access-6sl9p\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343363 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343337 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2af373-af3a-49d3-a022-9749ac02ce19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:08:48.343363 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343354 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-envoy\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343588 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343413 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-podinfo\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343588 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343461 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-certs\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343588 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343495 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-credential-socket\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343588 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343542 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-socket\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343588 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343556 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:08:48.343848 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343602 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-token\") pod \"9c2af373-af3a-49d3-a022-9749ac02ce19\" (UID: \"9c2af373-af3a-49d3-a022-9749ac02ce19\") " Aug 04 13:08:48.343848 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343739 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:08:48.343848 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343800 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:08:48.343987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343879 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:08:48.343987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343957 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.343987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343976 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.344100 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.343991 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.344100 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.344005 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2af373-af3a-49d3-a022-9749ac02ce19-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.344100 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.344018 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.345655 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.345625 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-kube-api-access-6sl9p" (OuterVolumeSpecName: "kube-api-access-6sl9p") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "kube-api-access-6sl9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:08:48.345988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.345964 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:08:48.346201 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.346181 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:08:48.346295 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.346183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c2af373-af3a-49d3-a022-9749ac02ce19" (UID: "9c2af373-af3a-49d3-a022-9749ac02ce19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:08:48.445156 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.445088 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sl9p\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-kube-api-access-6sl9p\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.445156 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.445147 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.445156 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.445157 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.445156 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.445165 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2af373-af3a-49d3-a022-9749ac02ce19-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:08:48.920648 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.920615 2578 generic.go:356] "Generic (PLEG): container finished" podID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerID="f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd" exitCode=0 Aug 04 13:08:48.920840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.920665 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" Aug 04 13:08:48.920840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.920703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" event={"ID":"9c2af373-af3a-49d3-a022-9749ac02ce19","Type":"ContainerDied","Data":"f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd"} Aug 04 13:08:48.920840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.920762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28" event={"ID":"9c2af373-af3a-49d3-a022-9749ac02ce19","Type":"ContainerDied","Data":"779aba562a3c77a861ba2978433c009f65193d2f578bf6ef4dd58e048181c345"} Aug 04 13:08:48.920840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.920793 2578 scope.go:117] "RemoveContainer" containerID="f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd" Aug 04 13:08:48.930767 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.930748 2578 scope.go:117] "RemoveContainer" containerID="94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f" Aug 04 13:08:48.939889 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.939866 2578 scope.go:117] "RemoveContainer" containerID="f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd" Aug 04 13:08:48.940313 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:08:48.940293 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd\": container with ID starting with f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd not found: ID does not exist" containerID="f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd" Aug 04 13:08:48.940384 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.940324 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd"} err="failed to get container status \"f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd\": rpc error: code = NotFound desc = could not find container \"f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd\": container with ID starting with f4f772830998583cdb187b207505032b1856031bbf09af2afee8c17ffc3a79dd not found: ID does not exist" Aug 04 13:08:48.940384 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.940344 2578 scope.go:117] "RemoveContainer" containerID="94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f" Aug 04 13:08:48.940619 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:08:48.940602 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f\": container with ID starting with 94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f not found: ID does not exist" containerID="94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f" Aug 04 13:08:48.940661 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.940626 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f"} err="failed to get container status \"94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f\": rpc error: code = NotFound desc = could not find container \"94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f\": container with ID starting with 94437c644af59de701b54a31ef5519d1696433611dea9531517d92522489947f not found: ID does not exist" Aug 04 13:08:48.959005 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.958970 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28"] Aug 04 13:08:48.964414 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:48.964382 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f758d543-openshift-default-545475f656-lnp28"] Aug 04 13:08:49.555399 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:49.555366 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" path="/var/lib/kubelet/pods/9c2af373-af3a-49d3-a022-9749ac02ce19/volumes" Aug 04 13:08:49.578439 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:49.578407 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:56.174991 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:56.174954 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:56.264006 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:56.263973 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:57.376261 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:57.376220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:58.973641 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:58.973592 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:08:59.073607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:08:59.073572 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:06.419285 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:06.419247 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:06.518658 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:06.518621 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:07.127823 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:07.127783 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:14.377475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:14.377435 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:14.484446 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:14.484408 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:14.581360 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:14.581311 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:25.233567 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:25.233531 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:25.514332 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:25.514222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:46.648584 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:46.648551 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:53.771825 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:53.771785 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:53.894326 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:53.894298 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:54.172638 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.172607 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:54.524530 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524436 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq"] Aug 04 13:09:54.524870 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524857 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc7e423-beff-46de-ba6c-8ef3cf441ab5" containerName="maas-api" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524872 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc7e423-beff-46de-ba6c-8ef3cf441ab5" containerName="maas-api" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524879 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524885 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524898 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524903 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524919 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d393b8f-12c3-419c-a5dc-3b3c287dafbf" containerName="payload-processing" Aug 04 13:09:54.524932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524924 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d393b8f-12c3-419c-a5dc-3b3c287dafbf" containerName="payload-processing" Aug 04 13:09:54.525191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524979 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" Aug 04 13:09:54.525191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524987 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2af373-af3a-49d3-a022-9749ac02ce19" containerName="istio-proxy" Aug 04 13:09:54.525191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.524994 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc7e423-beff-46de-ba6c-8ef3cf441ab5" containerName="maas-api" Aug 04 13:09:54.525191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.525003 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d393b8f-12c3-419c-a5dc-3b3c287dafbf" containerName="payload-processing" Aug 04 13:09:54.527731 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.527704 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:54.528259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.528244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.532594 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.532566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1c6080-openshift-default-dockercfg-qt5nh\"" Aug 04 13:09:54.532696 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.532613 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 13:09:54.543575 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.543548 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq"] Aug 04 13:09:54.675130 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-socket\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675307 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcb5e75-d521-4018-a86d-ec27789aa151-istiod-ca-cert\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675307 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-envoy\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675307 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-data\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675307 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-podinfo\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675307 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-token\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675579 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-certs\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675579 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-credential-socket\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.675579 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.675449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzgp\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-kube-api-access-8kzgp\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.685682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.685652 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:54.776788 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776706 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-socket\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.776788 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcb5e75-d521-4018-a86d-ec27789aa151-istiod-ca-cert\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-envoy\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-data\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-podinfo\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-token\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-certs\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-credential-socket\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.776955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzgp\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-kube-api-access-8kzgp\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.777231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-socket\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.777484 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-certs\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.777491 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-data\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.777553 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-credential-socket\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.777687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.777610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcb5e75-d521-4018-a86d-ec27789aa151-istiod-ca-cert\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.779344 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.779324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-envoy\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.779554 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.779536 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-podinfo\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.787475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.787434 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-token\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.787730 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.787708 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzgp\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-kube-api-access-8kzgp\") pod \"e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.838900 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.838859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:54.990496 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.990471 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq"] Aug 04 13:09:54.990884 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:09:54.990840 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcb5e75_d521_4018_a86d_ec27789aa151.slice/crio-23166bb589827e5498e528accf43f7ea734e15794f2b57a26bd4dae3ef566423 WatchSource:0}: Error finding container 23166bb589827e5498e528accf43f7ea734e15794f2b57a26bd4dae3ef566423: Status 404 returned error can't find the container with id 23166bb589827e5498e528accf43f7ea734e15794f2b57a26bd4dae3ef566423 Aug 04 13:09:54.993254 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.993223 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:09:54.993330 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.993287 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:09:54.993330 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:54.993317 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:09:55.255601 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:55.255563 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" event={"ID":"6bcb5e75-d521-4018-a86d-ec27789aa151","Type":"ContainerStarted","Data":"ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8"} Aug 04 13:09:55.255810 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:55.255615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" event={"ID":"6bcb5e75-d521-4018-a86d-ec27789aa151","Type":"ContainerStarted","Data":"23166bb589827e5498e528accf43f7ea734e15794f2b57a26bd4dae3ef566423"} Aug 04 13:09:55.277649 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:55.277595 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podStartSLOduration=1.27757842 podStartE2EDuration="1.27757842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:09:55.276556405 +0000 UTC m=+2526.232152202" watchObservedRunningTime="2026-08-04 13:09:55.27757842 +0000 UTC m=+2526.233174213" Aug 04 13:09:55.839552 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:55.839524 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:56.844357 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:56.844325 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:57.267227 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:57.267110 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:57.268093 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:57.268072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:09:57.445161 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:57.445102 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:57.539333 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:57.539243 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:58.533185 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.533143 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn"] Aug 04 13:09:58.537709 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.537676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.541506 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.541476 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1c6080-serving-cert\"" Aug 04 13:09:58.548667 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.548629 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn"] Aug 04 13:09:58.560547 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.560508 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:58.668999 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.668964 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:58.714845 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.714806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.714845 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.714852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.715073 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.714939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjp2\" (UniqueName: \"kubernetes.io/projected/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-kube-api-access-dzjp2\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.816050 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.815970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.816050 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.816007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.816258 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.816065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjp2\" (UniqueName: \"kubernetes.io/projected/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-kube-api-access-dzjp2\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.816697 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.816674 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.818403 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.818380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-maas-api-tls\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.827380 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.827357 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjp2\" (UniqueName: \"kubernetes.io/projected/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-kube-api-access-dzjp2\") pod \"maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:58.852374 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:58.852342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:09:59.006038 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.006009 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn"] Aug 04 13:09:59.007808 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:09:59.007778 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f27ac5d_172a_4bf2_ab3a_d75ae9bd8fe4.slice/crio-17fba19342dadd8ab390cdd37a8ca26a2daf90644f0624927a56189ece1e53b8 WatchSource:0}: Error finding container 17fba19342dadd8ab390cdd37a8ca26a2daf90644f0624927a56189ece1e53b8: Status 404 returned error can't find the container with id 17fba19342dadd8ab390cdd37a8ca26a2daf90644f0624927a56189ece1e53b8 Aug 04 13:09:59.126670 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.126641 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:59.226791 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.226759 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:09:59.256651 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.256619 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8"] Aug 04 13:09:59.261384 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.261363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.264474 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.264452 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1c6080\"" Aug 04 13:09:59.264645 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.264490 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1c6080-dockercfg-9l28p\"" Aug 04 13:09:59.270907 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.270885 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8"] Aug 04 13:09:59.278694 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.278665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" event={"ID":"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4","Type":"ContainerStarted","Data":"17fba19342dadd8ab390cdd37a8ca26a2daf90644f0624927a56189ece1e53b8"} Aug 04 13:09:59.320453 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.320412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1066a961-5d4a-4ac1-8b64-32c4f26478b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.320633 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.320514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dbc\" (UniqueName: \"kubernetes.io/projected/1066a961-5d4a-4ac1-8b64-32c4f26478b9-kube-api-access-q9dbc\") pod \"payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.421660 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.421575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1066a961-5d4a-4ac1-8b64-32c4f26478b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.421660 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.421622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dbc\" (UniqueName: \"kubernetes.io/projected/1066a961-5d4a-4ac1-8b64-32c4f26478b9-kube-api-access-q9dbc\") pod \"payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.422374 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.422354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1066a961-5d4a-4ac1-8b64-32c4f26478b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.434105 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.434080 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dbc\" (UniqueName: \"kubernetes.io/projected/1066a961-5d4a-4ac1-8b64-32c4f26478b9-kube-api-access-q9dbc\") pod \"payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.574787 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.574758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:09:59.733097 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:09:59.733014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8"] Aug 04 13:09:59.740911 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:09:59.740880 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1066a961_5d4a_4ac1_8b64_32c4f26478b9.slice/crio-cd80aeac850cd3acfbf59d2b1cf326f2d4c35eed8f7e318805f574ad70db43e1 WatchSource:0}: Error finding container cd80aeac850cd3acfbf59d2b1cf326f2d4c35eed8f7e318805f574ad70db43e1: Status 404 returned error can't find the container with id cd80aeac850cd3acfbf59d2b1cf326f2d4c35eed8f7e318805f574ad70db43e1 Aug 04 13:10:00.286053 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.286011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" event={"ID":"1066a961-5d4a-4ac1-8b64-32c4f26478b9","Type":"ContainerStarted","Data":"0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e"} Aug 04 13:10:00.286291 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.286079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" event={"ID":"1066a961-5d4a-4ac1-8b64-32c4f26478b9","Type":"ContainerStarted","Data":"cd80aeac850cd3acfbf59d2b1cf326f2d4c35eed8f7e318805f574ad70db43e1"} Aug 04 13:10:00.286291 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.286107 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:10:00.287475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.287449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" event={"ID":"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4","Type":"ContainerStarted","Data":"b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53"} Aug 04 13:10:00.287600 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.287567 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:10:00.304480 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.304424 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" podStartSLOduration=1.304408996 podStartE2EDuration="1.304408996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:10:00.303559004 +0000 UTC m=+2531.259154797" watchObservedRunningTime="2026-08-04 13:10:00.304408996 +0000 UTC m=+2531.260004767" Aug 04 13:10:00.325571 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:00.325514 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" podStartSLOduration=1.843845127 podStartE2EDuration="2.325494975s" totalImagesPullingTime="481.649848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:10:00.324305373 +0000 UTC m=+2531.279901199" watchObservedRunningTime="2026-08-04 13:10:00.325494975 +0000 UTC m=+2531.281090770" Aug 04 13:10:06.298868 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:06.298839 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:10:11.295961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:11.295925 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:10:12.269392 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.269314 2578 patch_prober.go:28] interesting pod/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Aug 04 13:10:12.269392 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.269374 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Aug 04 13:10:12.367487 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.367458 2578 generic.go:356] "Generic (PLEG): container finished" podID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerID="ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8" exitCode=0 Aug 04 13:10:12.367869 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.367535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" event={"ID":"6bcb5e75-d521-4018-a86d-ec27789aa151","Type":"ContainerDied","Data":"ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8"} Aug 04 13:10:12.367979 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.367952 2578 scope.go:117] "RemoveContainer" containerID="ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8" Aug 04 13:10:12.369005 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.368981 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:12.369057 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.369043 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:12.369113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:12.369073 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:13.375018 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:13.374979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" event={"ID":"6bcb5e75-d521-4018-a86d-ec27789aa151","Type":"ContainerStarted","Data":"341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a"} Aug 04 13:10:13.839436 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:13.839404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:10:13.839664 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:13.839643 2578 patch_prober.go:28] interesting pod/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Aug 04 13:10:13.839709 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:13.839689 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Aug 04 13:10:14.839789 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:14.839754 2578 patch_prober.go:28] interesting pod/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Aug 04 13:10:14.840228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:14.839819 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Aug 04 13:10:15.839498 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:15.839467 2578 patch_prober.go:28] interesting pod/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Aug 04 13:10:15.839668 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:15.839529 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Aug 04 13:10:16.840243 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:16.840193 2578 patch_prober.go:28] interesting pod/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Aug 04 13:10:16.840699 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:16.840266 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Aug 04 13:10:17.840212 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:17.840172 2578 patch_prober.go:28] interesting pod/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Aug 04 13:10:17.840470 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:17.840228 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Aug 04 13:10:18.843456 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:18.843419 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:10:18.843912 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:18.843812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:10:18.844619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:18.844601 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:10:22.454329 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:22.454295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:22.806889 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:22.806790 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:23.123356 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.123327 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz"] Aug 04 13:10:23.127300 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.127279 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.131255 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.131229 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3a8b01-openshift-default-dockercfg-x5jp2\"" Aug 04 13:10:23.139539 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.139516 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz"] Aug 04 13:10:23.154160 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.154108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:23.255230 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.255195 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:23.256306 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-envoy\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256363 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-data\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256410 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-socket\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-credential-socket\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256520 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-token\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256554 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4xg\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-kube-api-access-kj4xg\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256554 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-podinfo\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256651 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istiod-ca-cert\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.256762 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.256673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-certs\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357782 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-data\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-socket\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-credential-socket\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-token\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4xg\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-kube-api-access-kj4xg\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-podinfo\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.357988 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.357974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istiod-ca-cert\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-certs\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-envoy\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-data\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-socket\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358536 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358446 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-credential-socket\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358536 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-certs\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.358765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.358741 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istiod-ca-cert\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.360507 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.360485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-envoy\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.360724 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.360701 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-podinfo\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.369614 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.369583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-token\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.369982 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.369959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4xg\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-kube-api-access-kj4xg\") pod \"e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.438880 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.438798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:23.589609 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.589581 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz"] Aug 04 13:10:23.592230 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:10:23.592196 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b37d59_f33f_4727_b6c3_6b8ba7afbb74.slice/crio-ca4054c2519946ed6fdd2eb4b11ff632f1f9c34383ace9ddd1baa6622b83e343 WatchSource:0}: Error finding container ca4054c2519946ed6fdd2eb4b11ff632f1f9c34383ace9ddd1baa6622b83e343: Status 404 returned error can't find the container with id ca4054c2519946ed6fdd2eb4b11ff632f1f9c34383ace9ddd1baa6622b83e343 Aug 04 13:10:23.594483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.594451 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:23.594557 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.594518 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:23.594557 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:23.594550 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:24.432494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:24.432461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" event={"ID":"88b37d59-f33f-4727-b6c3-6b8ba7afbb74","Type":"ContainerStarted","Data":"b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29"} Aug 04 13:10:24.432682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:24.432511 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" event={"ID":"88b37d59-f33f-4727-b6c3-6b8ba7afbb74","Type":"ContainerStarted","Data":"ca4054c2519946ed6fdd2eb4b11ff632f1f9c34383ace9ddd1baa6622b83e343"} Aug 04 13:10:24.439467 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:24.439439 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:24.444459 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:24.444430 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:24.468834 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:24.468787 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podStartSLOduration=1.468771563 podStartE2EDuration="1.468771563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:10:24.467706368 +0000 UTC m=+2555.423302161" watchObservedRunningTime="2026-08-04 13:10:24.468771563 +0000 UTC m=+2555.424367356" Aug 04 13:10:25.438456 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:25.438419 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:25.439546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:25.439526 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:26.453745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:26.453712 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:26.555706 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:26.555671 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:27.729321 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:27.729283 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:27.936721 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:27.936685 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:28.283300 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:28.283262 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:28.387374 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:28.387334 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:10:40.440666 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.440576 2578 patch_prober.go:28] interesting pod/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Aug 04 13:10:40.440666 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.440632 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Aug 04 13:10:40.516017 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.515984 2578 generic.go:356] "Generic (PLEG): container finished" podID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerID="b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29" exitCode=0 Aug 04 13:10:40.516222 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.516058 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" event={"ID":"88b37d59-f33f-4727-b6c3-6b8ba7afbb74","Type":"ContainerDied","Data":"b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29"} Aug 04 13:10:40.516577 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.516561 2578 scope.go:117] "RemoveContainer" containerID="b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29" Aug 04 13:10:40.517702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.517678 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:40.517778 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.517743 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:40.517778 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:40.517773 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:10:41.526568 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:41.526527 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" event={"ID":"88b37d59-f33f-4727-b6c3-6b8ba7afbb74","Type":"ContainerStarted","Data":"dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e"} Aug 04 13:10:42.439235 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:42.439200 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:42.439501 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:42.439478 2578 patch_prober.go:28] interesting pod/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Aug 04 13:10:42.439587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:42.439531 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Aug 04 13:10:43.440243 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:43.440200 2578 patch_prober.go:28] interesting pod/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Aug 04 13:10:43.440674 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:43.440261 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Aug 04 13:10:44.439998 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:44.439957 2578 patch_prober.go:28] interesting pod/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Aug 04 13:10:44.440202 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:44.440022 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Aug 04 13:10:45.440099 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:45.440056 2578 patch_prober.go:28] interesting pod/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Aug 04 13:10:45.440551 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:45.440151 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Aug 04 13:10:46.439783 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:46.439740 2578 patch_prober.go:28] interesting pod/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Aug 04 13:10:46.439999 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:46.439807 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Aug 04 13:10:47.443042 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:47.442998 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:47.443453 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:47.443250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:47.443957 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:47.443943 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:10:51.486383 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:10:51.486349 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:11:04.543564 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:04.543529 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:11:22.004707 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.004578 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq"] Aug 04 13:11:22.007363 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.007346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.010514 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.010486 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3a8b01\"/\"openshift-service-ca.crt\"" Aug 04 13:11:22.011837 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.011804 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3a8b01\"/\"test-model-3a8b01-kserve-self-signed-certs\"" Aug 04 13:11:22.011837 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.011822 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3a8b01\"/\"kube-root-ca.crt\"" Aug 04 13:11:22.012041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.011917 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3a8b01\"/\"default-dockercfg-ddz9l\"" Aug 04 13:11:22.019149 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.019107 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq"] Aug 04 13:11:22.028402 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.028372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:11:22.045154 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.045083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-model-cache\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.045154 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.045153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-dshm\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.045400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.045226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-tmp-dir\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.045400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.045260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-home\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.045400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.045307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67334b9f-55ff-42aa-b384-d7f821331e79-tls-certs\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.045400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.045341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgn8w\" (UniqueName: \"kubernetes.io/projected/67334b9f-55ff-42aa-b384-d7f821331e79-kube-api-access-cgn8w\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.146642 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.146602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgn8w\" (UniqueName: \"kubernetes.io/projected/67334b9f-55ff-42aa-b384-d7f821331e79-kube-api-access-cgn8w\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.146821 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.146657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-model-cache\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.146821 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.146687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-dshm\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.146821 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.146724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-tmp-dir\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.146821 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.146768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-home\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.147022 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.146827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67334b9f-55ff-42aa-b384-d7f821331e79-tls-certs\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.147218 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.147183 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-model-cache\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.147351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.147197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-tmp-dir\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.147351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.147260 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-home\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.149309 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.149285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-dshm\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.149573 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.149548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67334b9f-55ff-42aa-b384-d7f821331e79-tls-certs\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.157186 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.157109 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgn8w\" (UniqueName: \"kubernetes.io/projected/67334b9f-55ff-42aa-b384-d7f821331e79-kube-api-access-cgn8w\") pod \"test-model-3a8b01-kserve-75c9896f79-l5wrq\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.317584 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.317488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:22.463903 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.463872 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq"] Aug 04 13:11:22.466268 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:11:22.466237 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67334b9f_55ff_42aa_b384_d7f821331e79.slice/crio-ed24649a6888428fed61f435f2c6dbfadf605c801bbe2deb70e03ce2d8355e7b WatchSource:0}: Error finding container ed24649a6888428fed61f435f2c6dbfadf605c801bbe2deb70e03ce2d8355e7b: Status 404 returned error can't find the container with id ed24649a6888428fed61f435f2c6dbfadf605c801bbe2deb70e03ce2d8355e7b Aug 04 13:11:22.751804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.751760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" event={"ID":"67334b9f-55ff-42aa-b384-d7f821331e79","Type":"ContainerStarted","Data":"ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a"} Aug 04 13:11:22.751983 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.751826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" event={"ID":"67334b9f-55ff-42aa-b384-d7f821331e79","Type":"ContainerStarted","Data":"ed24649a6888428fed61f435f2c6dbfadf605c801bbe2deb70e03ce2d8355e7b"} Aug 04 13:11:22.771427 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:22.771372 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" podStartSLOduration=1.7713497459999998 podStartE2EDuration="1.771349746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:11:22.770443627 +0000 UTC m=+2613.726039425" watchObservedRunningTime="2026-08-04 13:11:22.771349746 +0000 UTC m=+2613.726945540" Aug 04 13:11:24.178613 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:24.178577 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:11:32.318185 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:32.318100 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:32.318185 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:32.318200 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:32.331005 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:32.330978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:11:32.816098 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:11:32.816061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:12:01.304345 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:01.304308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:02.188956 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:02.188915 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq"] Aug 04 13:12:02.209338 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:02.209306 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:13.029617 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.029574 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn"] Aug 04 13:12:13.030231 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.029834 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" podUID="5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" containerName="maas-api" containerID="cri-o://b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53" gracePeriod=30 Aug 04 13:12:13.057337 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.057308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:13.090178 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.090134 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8"] Aug 04 13:12:13.090418 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.090373 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" podUID="1066a961-5d4a-4ac1-8b64-32c4f26478b9" containerName="payload-processing" containerID="cri-o://0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e" gracePeriod=30 Aug 04 13:12:13.156516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.156456 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:13.257483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.257394 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:13.322973 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.322945 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:13.345585 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.345558 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:12:13.374467 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.374439 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:12:13.461041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461006 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-openshift-service-ca\") pod \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " Aug 04 13:12:13.461264 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461067 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1066a961-5d4a-4ac1-8b64-32c4f26478b9-plugins-config-volume\") pod \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " Aug 04 13:12:13.461264 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461154 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-maas-api-tls\") pod \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " Aug 04 13:12:13.461264 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461185 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9dbc\" (UniqueName: \"kubernetes.io/projected/1066a961-5d4a-4ac1-8b64-32c4f26478b9-kube-api-access-q9dbc\") pod \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\" (UID: \"1066a961-5d4a-4ac1-8b64-32c4f26478b9\") " Aug 04 13:12:13.461264 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461223 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzjp2\" (UniqueName: \"kubernetes.io/projected/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-kube-api-access-dzjp2\") pod \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\" (UID: \"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4\") " Aug 04 13:12:13.461503 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461480 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" (UID: "5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:12:13.461564 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.461534 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1066a961-5d4a-4ac1-8b64-32c4f26478b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1066a961-5d4a-4ac1-8b64-32c4f26478b9" (UID: "1066a961-5d4a-4ac1-8b64-32c4f26478b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:12:13.463464 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.463434 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" (UID: "5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:12:13.463575 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.463487 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-kube-api-access-dzjp2" (OuterVolumeSpecName: "kube-api-access-dzjp2") pod "5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" (UID: "5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4"). InnerVolumeSpecName "kube-api-access-dzjp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:13.463575 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.463498 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1066a961-5d4a-4ac1-8b64-32c4f26478b9-kube-api-access-q9dbc" (OuterVolumeSpecName: "kube-api-access-q9dbc") pod "1066a961-5d4a-4ac1-8b64-32c4f26478b9" (UID: "1066a961-5d4a-4ac1-8b64-32c4f26478b9"). InnerVolumeSpecName "kube-api-access-q9dbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:13.561961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.561884 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-maas-api-tls\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:13.561961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.561910 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9dbc\" (UniqueName: \"kubernetes.io/projected/1066a961-5d4a-4ac1-8b64-32c4f26478b9-kube-api-access-q9dbc\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:13.561961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.561921 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjp2\" (UniqueName: \"kubernetes.io/projected/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-kube-api-access-dzjp2\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:13.561961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.561929 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4-openshift-service-ca\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:13.561961 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.561938 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1066a961-5d4a-4ac1-8b64-32c4f26478b9-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:13.682858 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:13.682817 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:14.020689 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.020650 2578 generic.go:356] "Generic (PLEG): container finished" podID="5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" containerID="b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53" exitCode=0 Aug 04 13:12:14.020949 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.020724 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" Aug 04 13:12:14.020949 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.020738 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" event={"ID":"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4","Type":"ContainerDied","Data":"b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53"} Aug 04 13:12:14.020949 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.020794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn" event={"ID":"5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4","Type":"ContainerDied","Data":"17fba19342dadd8ab390cdd37a8ca26a2daf90644f0624927a56189ece1e53b8"} Aug 04 13:12:14.020949 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.020827 2578 scope.go:117] "RemoveContainer" containerID="b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53" Aug 04 13:12:14.022194 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.022169 2578 generic.go:356] "Generic (PLEG): container finished" podID="1066a961-5d4a-4ac1-8b64-32c4f26478b9" containerID="0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e" exitCode=0 Aug 04 13:12:14.022325 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.022202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" event={"ID":"1066a961-5d4a-4ac1-8b64-32c4f26478b9","Type":"ContainerDied","Data":"0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e"} Aug 04 13:12:14.022325 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.022231 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" Aug 04 13:12:14.022325 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.022240 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8" event={"ID":"1066a961-5d4a-4ac1-8b64-32c4f26478b9","Type":"ContainerDied","Data":"cd80aeac850cd3acfbf59d2b1cf326f2d4c35eed8f7e318805f574ad70db43e1"} Aug 04 13:12:14.029925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.029906 2578 scope.go:117] "RemoveContainer" containerID="b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53" Aug 04 13:12:14.030263 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:14.030197 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53\": container with ID starting with b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53 not found: ID does not exist" containerID="b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53" Aug 04 13:12:14.030263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.030224 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53"} err="failed to get container status \"b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53\": rpc error: code = NotFound desc = could not find container \"b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53\": container with ID starting with b2c70de01e63f46860653e3ebef015a0cf3bf827aa620f720cc6196d41250d53 not found: ID does not exist" Aug 04 13:12:14.030263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.030240 2578 scope.go:117] "RemoveContainer" containerID="0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e" Aug 04 13:12:14.038385 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.038235 2578 scope.go:117] "RemoveContainer" containerID="0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e" Aug 04 13:12:14.038572 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:14.038552 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e\": container with ID starting with 0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e not found: ID does not exist" containerID="0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e" Aug 04 13:12:14.038647 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.038580 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e"} err="failed to get container status \"0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e\": rpc error: code = NotFound desc = could not find container \"0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e\": container with ID starting with 0ecfdb1674ec5fa5bf4452e8f83aa24bd18be97572ee1e30805c7c6239acb58e not found: ID does not exist" Aug 04 13:12:14.040965 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.040935 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8"] Aug 04 13:12:14.045374 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.045345 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1c6080-5997c5854-gcgg8"] Aug 04 13:12:14.057534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.057509 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn"] Aug 04 13:12:14.062375 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:14.062351 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1c6080-77747cfb4c-qrvrn"] Aug 04 13:12:15.554986 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:15.554952 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1066a961-5d4a-4ac1-8b64-32c4f26478b9" path="/var/lib/kubelet/pods/1066a961-5d4a-4ac1-8b64-32c4f26478b9/volumes" Aug 04 13:12:15.555508 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:15.555433 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" path="/var/lib/kubelet/pods/5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4/volumes" Aug 04 13:12:17.209341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.209273 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" podUID="67334b9f-55ff-42aa-b384-d7f821331e79" containerName="main" containerID="cri-o://ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a" gracePeriod=45 Aug 04 13:12:17.458667 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.458631 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:12:17.603937 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.603899 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-model-cache\") pod \"67334b9f-55ff-42aa-b384-d7f821331e79\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " Aug 04 13:12:17.604142 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.603955 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-tmp-dir\") pod \"67334b9f-55ff-42aa-b384-d7f821331e79\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " Aug 04 13:12:17.604142 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604005 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-dshm\") pod \"67334b9f-55ff-42aa-b384-d7f821331e79\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " Aug 04 13:12:17.604142 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-home\") pod \"67334b9f-55ff-42aa-b384-d7f821331e79\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " Aug 04 13:12:17.604142 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604104 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67334b9f-55ff-42aa-b384-d7f821331e79-tls-certs\") pod \"67334b9f-55ff-42aa-b384-d7f821331e79\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " Aug 04 13:12:17.604370 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604166 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgn8w\" (UniqueName: \"kubernetes.io/projected/67334b9f-55ff-42aa-b384-d7f821331e79-kube-api-access-cgn8w\") pod \"67334b9f-55ff-42aa-b384-d7f821331e79\" (UID: \"67334b9f-55ff-42aa-b384-d7f821331e79\") " Aug 04 13:12:17.604370 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604271 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67334b9f-55ff-42aa-b384-d7f821331e79" (UID: "67334b9f-55ff-42aa-b384-d7f821331e79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:17.604370 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604287 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-model-cache" (OuterVolumeSpecName: "model-cache") pod "67334b9f-55ff-42aa-b384-d7f821331e79" (UID: "67334b9f-55ff-42aa-b384-d7f821331e79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:17.604518 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604379 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-home" (OuterVolumeSpecName: "home") pod "67334b9f-55ff-42aa-b384-d7f821331e79" (UID: "67334b9f-55ff-42aa-b384-d7f821331e79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:17.604597 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604577 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-tmp-dir\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:17.604650 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604604 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-home\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:17.604650 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.604619 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-model-cache\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:17.606590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.606564 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67334b9f-55ff-42aa-b384-d7f821331e79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67334b9f-55ff-42aa-b384-d7f821331e79" (UID: "67334b9f-55ff-42aa-b384-d7f821331e79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:12:17.606590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.606578 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67334b9f-55ff-42aa-b384-d7f821331e79-kube-api-access-cgn8w" (OuterVolumeSpecName: "kube-api-access-cgn8w") pod "67334b9f-55ff-42aa-b384-d7f821331e79" (UID: "67334b9f-55ff-42aa-b384-d7f821331e79"). InnerVolumeSpecName "kube-api-access-cgn8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:17.606743 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.606621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-dshm" (OuterVolumeSpecName: "dshm") pod "67334b9f-55ff-42aa-b384-d7f821331e79" (UID: "67334b9f-55ff-42aa-b384-d7f821331e79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:17.705325 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.705284 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67334b9f-55ff-42aa-b384-d7f821331e79-dshm\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:17.705325 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.705318 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67334b9f-55ff-42aa-b384-d7f821331e79-tls-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:17.705325 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:17.705330 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgn8w\" (UniqueName: \"kubernetes.io/projected/67334b9f-55ff-42aa-b384-d7f821331e79-kube-api-access-cgn8w\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:18.044544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.044462 2578 generic.go:356] "Generic (PLEG): container finished" podID="67334b9f-55ff-42aa-b384-d7f821331e79" containerID="ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a" exitCode=0 Aug 04 13:12:18.044544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.044501 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" event={"ID":"67334b9f-55ff-42aa-b384-d7f821331e79","Type":"ContainerDied","Data":"ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a"} Aug 04 13:12:18.044544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.044506 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" Aug 04 13:12:18.044544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.044531 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq" event={"ID":"67334b9f-55ff-42aa-b384-d7f821331e79","Type":"ContainerDied","Data":"ed24649a6888428fed61f435f2c6dbfadf605c801bbe2deb70e03ce2d8355e7b"} Aug 04 13:12:18.044544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.044547 2578 scope.go:117] "RemoveContainer" containerID="ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a" Aug 04 13:12:18.054114 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.054094 2578 scope.go:117] "RemoveContainer" containerID="ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a" Aug 04 13:12:18.054460 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:18.054439 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a\": container with ID starting with ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a not found: ID does not exist" containerID="ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a" Aug 04 13:12:18.054546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.054469 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a"} err="failed to get container status \"ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a\": rpc error: code = NotFound desc = could not find container \"ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a\": container with ID starting with ede3c97cee6a2d8f1ff834a4da978d227347989c3a41be07c137b03454c1c91a not found: ID does not exist" Aug 04 13:12:18.067742 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.067715 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq"] Aug 04 13:12:18.072356 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:18.072333 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3a8b01/test-model-3a8b01-kserve-75c9896f79-l5wrq"] Aug 04 13:12:19.558497 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:19.558462 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67334b9f-55ff-42aa-b384-d7f821331e79" path="/var/lib/kubelet/pods/67334b9f-55ff-42aa-b384-d7f821331e79/volumes" Aug 04 13:12:20.153404 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:20.153365 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:20.259204 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:20.259165 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:20.414083 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:20.413995 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq"] Aug 04 13:12:20.414265 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:20.414228 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" containerID="cri-o://341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a" gracePeriod=30 Aug 04 13:12:20.438565 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:20.438489 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:20.689509 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:20.689420 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:25.669619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.669589 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:12:25.783257 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783176 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-podinfo\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783257 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783228 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-envoy\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783265 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-token\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783312 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-data\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783352 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-certs\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783386 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kzgp\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-kube-api-access-8kzgp\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783408 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-credential-socket\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783428 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-socket\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783763 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783564 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcb5e75-d521-4018-a86d-ec27789aa151-istiod-ca-cert\") pod \"6bcb5e75-d521-4018-a86d-ec27789aa151\" (UID: \"6bcb5e75-d521-4018-a86d-ec27789aa151\") " Aug 04 13:12:25.783763 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783664 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-data" (OuterVolumeSpecName: "istio-data") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:25.783763 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783701 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:25.783906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783860 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:25.783906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783873 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:25.784008 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783980 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.784008 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783999 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.784080 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.784015 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.784080 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.784028 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.784080 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.783980 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcb5e75-d521-4018-a86d-ec27789aa151-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:12:25.785539 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.785506 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:12:25.785963 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.785941 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-kube-api-access-8kzgp" (OuterVolumeSpecName: "kube-api-access-8kzgp") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "kube-api-access-8kzgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:25.786228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.786213 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:25.786296 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.786223 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-token" (OuterVolumeSpecName: "istio-token") pod "6bcb5e75-d521-4018-a86d-ec27789aa151" (UID: "6bcb5e75-d521-4018-a86d-ec27789aa151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:25.885336 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.885298 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bcb5e75-d521-4018-a86d-ec27789aa151-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.885336 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.885334 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.885336 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.885343 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.885559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.885351 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:25.885559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:25.885359 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzgp\" (UniqueName: \"kubernetes.io/projected/6bcb5e75-d521-4018-a86d-ec27789aa151-kube-api-access-8kzgp\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:26.090852 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.090820 2578 generic.go:356] "Generic (PLEG): container finished" podID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerID="341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a" exitCode=0 Aug 04 13:12:26.091030 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.090865 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" Aug 04 13:12:26.091030 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.090909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" event={"ID":"6bcb5e75-d521-4018-a86d-ec27789aa151","Type":"ContainerDied","Data":"341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a"} Aug 04 13:12:26.091030 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.090973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq" event={"ID":"6bcb5e75-d521-4018-a86d-ec27789aa151","Type":"ContainerDied","Data":"23166bb589827e5498e528accf43f7ea734e15794f2b57a26bd4dae3ef566423"} Aug 04 13:12:26.091030 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.091002 2578 scope.go:117] "RemoveContainer" containerID="341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a" Aug 04 13:12:26.100446 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.100430 2578 scope.go:117] "RemoveContainer" containerID="ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8" Aug 04 13:12:26.108680 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.108663 2578 scope.go:117] "RemoveContainer" containerID="341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a" Aug 04 13:12:26.108956 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:26.108939 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a\": container with ID starting with 341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a not found: ID does not exist" containerID="341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a" Aug 04 13:12:26.108995 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.108966 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a"} err="failed to get container status \"341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a\": rpc error: code = NotFound desc = could not find container \"341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a\": container with ID starting with 341c9854f75af879ab22a1d9198da8a58d5f71fbae9721f3512503dd86b7906a not found: ID does not exist" Aug 04 13:12:26.108995 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.108982 2578 scope.go:117] "RemoveContainer" containerID="ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8" Aug 04 13:12:26.109220 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:26.109204 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8\": container with ID starting with ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8 not found: ID does not exist" containerID="ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8" Aug 04 13:12:26.109265 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.109225 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8"} err="failed to get container status \"ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8\": rpc error: code = NotFound desc = could not find container \"ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8\": container with ID starting with ad253798a4d61bc76cf6fbd80f08a0c0a5f820cc3ef20895cf89d5ce526959a8 not found: ID does not exist" Aug 04 13:12:26.115283 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.115255 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq"] Aug 04 13:12:26.119828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:26.119802 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1c6080-openshift-default-779f9f6d58-sc2mq"] Aug 04 13:12:27.553054 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:27.553019 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" path="/var/lib/kubelet/pods/6bcb5e75-d521-4018-a86d-ec27789aa151/volumes" Aug 04 13:12:31.319024 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:31.318921 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:31.406529 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:31.406498 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:31.470195 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:31.470100 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:31.536619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:31.536587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:31.942328 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:31.942291 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:38.290669 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:38.290627 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:38.351763 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:38.351722 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:38.540616 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:38.540580 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz"] Aug 04 13:12:38.540893 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:38.540835 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" containerID="cri-o://dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e" gracePeriod=30 Aug 04 13:12:38.563102 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:38.563064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:38.815382 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:38.815303 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:39.136702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.136644 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:39.447039 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.446961 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9"] Aug 04 13:12:39.447475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447456 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" Aug 04 13:12:39.447555 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447478 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" Aug 04 13:12:39.447555 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447495 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67334b9f-55ff-42aa-b384-d7f821331e79" containerName="main" Aug 04 13:12:39.447555 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447504 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67334b9f-55ff-42aa-b384-d7f821331e79" containerName="main" Aug 04 13:12:39.447555 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447542 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1066a961-5d4a-4ac1-8b64-32c4f26478b9" containerName="payload-processing" Aug 04 13:12:39.447555 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447552 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1066a961-5d4a-4ac1-8b64-32c4f26478b9" containerName="payload-processing" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447568 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" containerName="maas-api" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447577 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" containerName="maas-api" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447675 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447690 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1066a961-5d4a-4ac1-8b64-32c4f26478b9" containerName="payload-processing" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447701 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447713 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67334b9f-55ff-42aa-b384-d7f821331e79" containerName="main" Aug 04 13:12:39.447798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.447725 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f27ac5d-172a-4bf2-ab3a-d75ae9bd8fe4" containerName="maas-api" Aug 04 13:12:39.452498 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.452479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.455304 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.455285 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0c5286-openshift-default-dockercfg-x9whc\"" Aug 04 13:12:39.463850 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.463830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9"] Aug 04 13:12:39.473468 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.473442 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:39.510231 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-data\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-podinfo\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istiod-ca-cert\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-certs\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9h8f\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-kube-api-access-v9h8f\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-credential-socket\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-envoy\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-socket\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.510663 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.510493 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-token\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611374 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-token\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-data\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-podinfo\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istiod-ca-cert\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-certs\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9h8f\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-kube-api-access-v9h8f\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-credential-socket\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-envoy\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.611906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.611705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-socket\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.612178 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.612151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-socket\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.612442 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.612419 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-data\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.612856 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.612813 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-credential-socket\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.612951 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.612935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-certs\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.613057 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.613040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istiod-ca-cert\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.614552 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.614520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-envoy\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.614872 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.614852 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-podinfo\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.621424 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.621405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-token\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.621674 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.621650 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9h8f\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-kube-api-access-v9h8f\") pod \"e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.763972 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.763877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:39.907919 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.907894 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9"] Aug 04 13:12:39.910676 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:12:39.910648 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71e549b_d76d_4ce0_bb7c_413e7abbf819.slice/crio-927caabd837d38150650f75db9db835b70c8bba7b7f27e043d26f730583c0d7f WatchSource:0}: Error finding container 927caabd837d38150650f75db9db835b70c8bba7b7f27e043d26f730583c0d7f: Status 404 returned error can't find the container with id 927caabd837d38150650f75db9db835b70c8bba7b7f27e043d26f730583c0d7f Aug 04 13:12:39.912849 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.912821 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:12:39.912933 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.912884 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:12:39.912976 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.912933 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:12:39.971538 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:39.971509 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:40.171288 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:40.171250 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" event={"ID":"f71e549b-d76d-4ce0-bb7c-413e7abbf819","Type":"ContainerStarted","Data":"195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c"} Aug 04 13:12:40.171494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:40.171329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" event={"ID":"f71e549b-d76d-4ce0-bb7c-413e7abbf819","Type":"ContainerStarted","Data":"927caabd837d38150650f75db9db835b70c8bba7b7f27e043d26f730583c0d7f"} Aug 04 13:12:40.193415 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:40.193354 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podStartSLOduration=1.193334482 podStartE2EDuration="1.193334482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:12:40.193330172 +0000 UTC m=+2691.148925967" watchObservedRunningTime="2026-08-04 13:12:40.193334482 +0000 UTC m=+2691.148930279" Aug 04 13:12:40.765101 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:40.765065 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:40.770018 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:40.769991 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:41.177262 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:41.177226 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:41.178209 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:41.178190 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:42.128315 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:42.128278 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:42.230817 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:42.230787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:43.286359 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.286319 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:43.488534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.488501 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:43.796154 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.796113 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:12:43.803064 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.803000 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:43.856784 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.856752 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-credential-socket\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.856944 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.856821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-envoy\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.856944 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.856852 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-data\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.856945 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istiod-ca-cert\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.856971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-podinfo\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857013 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-token\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857039 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj4xg\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-kube-api-access-kj4xg\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857066 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-certs\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857065 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:43.857191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-socket\") pod \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\" (UID: \"88b37d59-f33f-4727-b6c3-6b8ba7afbb74\") " Aug 04 13:12:43.857191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857163 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-data" (OuterVolumeSpecName: "istio-data") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:43.857472 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857332 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:43.857472 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857392 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:12:43.857591 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857506 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:43.857829 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857776 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.857829 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857801 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.857829 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857814 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.857829 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857826 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.858073 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.857839 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.859316 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.859291 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:12:43.859430 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.859379 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-token" (OuterVolumeSpecName: "istio-token") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:43.859801 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.859778 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-kube-api-access-kj4xg" (OuterVolumeSpecName: "kube-api-access-kj4xg") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "kube-api-access-kj4xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:12:43.859873 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.859774 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88b37d59-f33f-4727-b6c3-6b8ba7afbb74" (UID: "88b37d59-f33f-4727-b6c3-6b8ba7afbb74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:12:43.899082 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.899053 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:12:43.959019 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.958986 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.959019 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.959010 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.959019 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.959019 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4xg\" (UniqueName: \"kubernetes.io/projected/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-kube-api-access-kj4xg\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.959019 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.959028 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b37d59-f33f-4727-b6c3-6b8ba7afbb74-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:12:43.991718 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.991685 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb"] Aug 04 13:12:43.992215 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992176 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" Aug 04 13:12:43.992215 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992199 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bcb5e75-d521-4018-a86d-ec27789aa151" containerName="istio-proxy" Aug 04 13:12:43.992400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992224 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" Aug 04 13:12:43.992400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992235 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" Aug 04 13:12:43.992400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992251 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" Aug 04 13:12:43.992400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992260 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" Aug 04 13:12:43.992616 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992401 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" Aug 04 13:12:43.992616 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.992418 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerName="istio-proxy" Aug 04 13:12:43.995969 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.995949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:43.999351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.999329 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c5286-dockercfg-9rd2n\"" Aug 04 13:12:43.999351 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:43.999350 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c5286\"" Aug 04 13:12:44.006312 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.006287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb"] Aug 04 13:12:44.060546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.060472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879bj\" (UniqueName: \"kubernetes.io/projected/20e6976e-1893-4ed9-8fd8-e5c47b297c81-kube-api-access-879bj\") pod \"payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.060546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.060516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e6976e-1893-4ed9-8fd8-e5c47b297c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.161560 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.161526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879bj\" (UniqueName: \"kubernetes.io/projected/20e6976e-1893-4ed9-8fd8-e5c47b297c81-kube-api-access-879bj\") pod \"payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.161734 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.161569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e6976e-1893-4ed9-8fd8-e5c47b297c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.162423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.162403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e6976e-1893-4ed9-8fd8-e5c47b297c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.170653 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.170628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-879bj\" (UniqueName: \"kubernetes.io/projected/20e6976e-1893-4ed9-8fd8-e5c47b297c81-kube-api-access-879bj\") pod \"payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.200786 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.200756 2578 generic.go:356] "Generic (PLEG): container finished" podID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" containerID="dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e" exitCode=0 Aug 04 13:12:44.200942 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.200807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" event={"ID":"88b37d59-f33f-4727-b6c3-6b8ba7afbb74","Type":"ContainerDied","Data":"dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e"} Aug 04 13:12:44.200942 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.200874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" event={"ID":"88b37d59-f33f-4727-b6c3-6b8ba7afbb74","Type":"ContainerDied","Data":"ca4054c2519946ed6fdd2eb4b11ff632f1f9c34383ace9ddd1baa6622b83e343"} Aug 04 13:12:44.200942 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.200875 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz" Aug 04 13:12:44.200942 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.200899 2578 scope.go:117] "RemoveContainer" containerID="dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e" Aug 04 13:12:44.210014 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.209995 2578 scope.go:117] "RemoveContainer" containerID="b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29" Aug 04 13:12:44.219283 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.219259 2578 scope.go:117] "RemoveContainer" containerID="dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e" Aug 04 13:12:44.219560 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:44.219538 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e\": container with ID starting with dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e not found: ID does not exist" containerID="dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e" Aug 04 13:12:44.219648 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.219572 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e"} err="failed to get container status \"dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e\": rpc error: code = NotFound desc = could not find container \"dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e\": container with ID starting with dc1d60e0d9fac0a0133cfb69fda1ee87d04d6e455210c5de74fc384de752cb2e not found: ID does not exist" Aug 04 13:12:44.219648 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.219596 2578 scope.go:117] "RemoveContainer" containerID="b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29" Aug 04 13:12:44.219881 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:12:44.219865 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29\": container with ID starting with b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29 not found: ID does not exist" containerID="b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29" Aug 04 13:12:44.219935 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.219888 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29"} err="failed to get container status \"b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29\": rpc error: code = NotFound desc = could not find container \"b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29\": container with ID starting with b2f5d97d49d22d920afbed866a9e8fa7729720d006c78f272d8a198d7e836f29 not found: ID does not exist" Aug 04 13:12:44.224482 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.224459 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz"] Aug 04 13:12:44.231347 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.231323 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3a8b01-openshift-default-659b7f68cd-qbfkz"] Aug 04 13:12:44.307331 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.307297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:44.448442 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:44.448412 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb"] Aug 04 13:12:44.450670 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:12:44.450646 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e6976e_1893_4ed9_8fd8_e5c47b297c81.slice/crio-c9974f4428f64548e559a5d6b87206a489a690472600ea1c4c725f0dfd858313 WatchSource:0}: Error finding container c9974f4428f64548e559a5d6b87206a489a690472600ea1c4c725f0dfd858313: Status 404 returned error can't find the container with id c9974f4428f64548e559a5d6b87206a489a690472600ea1c4c725f0dfd858313 Aug 04 13:12:45.207407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:45.207370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" event={"ID":"20e6976e-1893-4ed9-8fd8-e5c47b297c81","Type":"ContainerStarted","Data":"be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f"} Aug 04 13:12:45.207638 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:45.207443 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:45.207638 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:45.207469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" event={"ID":"20e6976e-1893-4ed9-8fd8-e5c47b297c81","Type":"ContainerStarted","Data":"c9974f4428f64548e559a5d6b87206a489a690472600ea1c4c725f0dfd858313"} Aug 04 13:12:45.228763 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:45.228716 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" podStartSLOduration=2.228699255 podStartE2EDuration="2.228699255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:12:45.226406731 +0000 UTC m=+2696.182002536" watchObservedRunningTime="2026-08-04 13:12:45.228699255 +0000 UTC m=+2696.184295049" Aug 04 13:12:45.554067 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:45.553987 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b37d59-f33f-4727-b6c3-6b8ba7afbb74" path="/var/lib/kubelet/pods/88b37d59-f33f-4727-b6c3-6b8ba7afbb74/volumes" Aug 04 13:12:49.653829 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:49.653802 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:12:49.662172 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:49.662143 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:12:55.264755 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:55.264630 2578 generic.go:356] "Generic (PLEG): container finished" podID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerID="195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c" exitCode=0 Aug 04 13:12:55.264755 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:55.264710 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" event={"ID":"f71e549b-d76d-4ce0-bb7c-413e7abbf819","Type":"ContainerDied","Data":"195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c"} Aug 04 13:12:55.265262 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:55.265203 2578 scope.go:117] "RemoveContainer" containerID="195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c" Aug 04 13:12:55.266343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:55.266319 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:12:55.266414 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:55.266383 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:12:55.266457 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:55.266413 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:12:56.178007 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:56.177964 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:56.218067 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:56.217161 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:12:56.272771 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:56.272740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" event={"ID":"f71e549b-d76d-4ce0-bb7c-413e7abbf819","Type":"ContainerStarted","Data":"95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85"} Aug 04 13:12:56.764798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:56.764762 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:12:56.765872 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:56.765847 2578 patch_prober.go:28] interesting pod/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Aug 04 13:12:56.765963 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:56.765895 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Aug 04 13:12:57.764868 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:57.764832 2578 patch_prober.go:28] interesting pod/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Aug 04 13:12:57.765288 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:57.764892 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Aug 04 13:12:58.764492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:58.764454 2578 patch_prober.go:28] interesting pod/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Aug 04 13:12:58.764671 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:58.764507 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Aug 04 13:12:59.764946 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:59.764902 2578 patch_prober.go:28] interesting pod/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Aug 04 13:12:59.765458 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:12:59.764985 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Aug 04 13:13:00.764850 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:00.764815 2578 patch_prober.go:28] interesting pod/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Aug 04 13:13:00.765043 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:00.764875 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Aug 04 13:13:01.768458 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:01.768423 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:13:01.768899 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:01.768677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:13:01.769488 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:01.769469 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:13:07.144627 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:07.144587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:07.648434 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:07.648400 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:07.934649 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:07.934569 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s"] Aug 04 13:13:07.938453 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:07.938434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:07.942688 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:07.942667 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c1a14d-openshift-default-dockercfg-frmsh\"" Aug 04 13:13:07.951160 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:07.951134 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s"] Aug 04 13:13:08.003631 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.003591 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:08.005865 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.005843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-data\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.005947 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.005907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-certs\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.005994 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.005953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c64759-f729-4c81-8358-5e6128959dd3-istio-podinfo\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.006030 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.005990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-istio-token\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.006065 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.006043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-envoy\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.006099 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.006080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-socket\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.006219 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.006101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzq69\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-kube-api-access-mzq69\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.006219 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.006139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-credential-socket\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.006219 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.006175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c64759-f729-4c81-8358-5e6128959dd3-istiod-ca-cert\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107408 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-envoy\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107408 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-socket\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107690 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzq69\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-kube-api-access-mzq69\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107690 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-credential-socket\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107690 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c64759-f729-4c81-8358-5e6128959dd3-istiod-ca-cert\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107690 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-data\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-certs\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c64759-f729-4c81-8358-5e6128959dd3-istio-podinfo\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-socket\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-istio-token\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.107902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.107875 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-credential-socket\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.108193 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.108055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-data\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.108193 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.108104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-certs\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.108410 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.108374 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c64759-f729-4c81-8358-5e6128959dd3-istiod-ca-cert\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.109894 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.109874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-envoy\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.110455 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.110433 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c64759-f729-4c81-8358-5e6128959dd3-istio-podinfo\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.118072 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.118039 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzq69\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-kube-api-access-mzq69\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.118500 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.118075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-istio-token\") pod \"e2e-api-b-c1a14d-openshift-default-68557657fd-6579s\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.249101 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.249008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:08.398031 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.398007 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s"] Aug 04 13:13:08.400603 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:13:08.400569 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c64759_f729_4c81_8358_5e6128959dd3.slice/crio-8e1289564f5c5817a6f9a7716032078a6a49272f973ce12636cd873d35e2889c WatchSource:0}: Error finding container 8e1289564f5c5817a6f9a7716032078a6a49272f973ce12636cd873d35e2889c: Status 404 returned error can't find the container with id 8e1289564f5c5817a6f9a7716032078a6a49272f973ce12636cd873d35e2889c Aug 04 13:13:08.402955 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.402917 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:13:08.403060 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.403008 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:13:08.403060 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.403055 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:13:08.864058 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:08.864015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:09.343491 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:09.343463 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" event={"ID":"06c64759-f729-4c81-8358-5e6128959dd3","Type":"ContainerStarted","Data":"c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223"} Aug 04 13:13:09.343937 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:09.343510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" event={"ID":"06c64759-f729-4c81-8358-5e6128959dd3","Type":"ContainerStarted","Data":"8e1289564f5c5817a6f9a7716032078a6a49272f973ce12636cd873d35e2889c"} Aug 04 13:13:09.367557 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:09.367505 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podStartSLOduration=2.367489798 podStartE2EDuration="2.367489798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:13:09.364737445 +0000 UTC m=+2720.320333263" watchObservedRunningTime="2026-08-04 13:13:09.367489798 +0000 UTC m=+2720.323085591" Aug 04 13:13:10.249444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:10.249415 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:10.254750 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:10.254724 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:10.355598 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:10.355550 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:10.356740 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:10.356717 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:11.350095 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:11.350058 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:11.443680 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:11.443643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:12.502469 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:12.502437 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:13.094520 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.094483 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:13.196785 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.196750 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:13.285420 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.285386 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd"] Aug 04 13:13:13.289186 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.289167 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.292341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.292314 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c1a14d\"" Aug 04 13:13:13.292495 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.292378 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c1a14d-dockercfg-mvzks\"" Aug 04 13:13:13.300177 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.300149 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd"] Aug 04 13:13:13.360908 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.360806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441a6eed-f13d-43e1-b3a2-597df55a29cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.360908 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.360898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djnw\" (UniqueName: \"kubernetes.io/projected/441a6eed-f13d-43e1-b3a2-597df55a29cd-kube-api-access-7djnw\") pod \"payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.462294 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.462260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djnw\" (UniqueName: \"kubernetes.io/projected/441a6eed-f13d-43e1-b3a2-597df55a29cd-kube-api-access-7djnw\") pod \"payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.462492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.462357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441a6eed-f13d-43e1-b3a2-597df55a29cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.463056 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.463034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441a6eed-f13d-43e1-b3a2-597df55a29cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.478612 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.478585 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djnw\" (UniqueName: \"kubernetes.io/projected/441a6eed-f13d-43e1-b3a2-597df55a29cd-kube-api-access-7djnw\") pod \"payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.601702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.601666 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:13.750868 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:13.750790 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd"] Aug 04 13:13:13.753825 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:13:13.753799 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441a6eed_f13d_43e1_b3a2_597df55a29cd.slice/crio-61d9f82e12c50830dace2c6a5fc28d52ef52312609187f59d2c4a3ba1588f8ee WatchSource:0}: Error finding container 61d9f82e12c50830dace2c6a5fc28d52ef52312609187f59d2c4a3ba1588f8ee: Status 404 returned error can't find the container with id 61d9f82e12c50830dace2c6a5fc28d52ef52312609187f59d2c4a3ba1588f8ee Aug 04 13:13:14.377663 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:14.377628 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" event={"ID":"441a6eed-f13d-43e1-b3a2-597df55a29cd","Type":"ContainerStarted","Data":"0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75"} Aug 04 13:13:14.377876 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:14.377679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" event={"ID":"441a6eed-f13d-43e1-b3a2-597df55a29cd","Type":"ContainerStarted","Data":"61d9f82e12c50830dace2c6a5fc28d52ef52312609187f59d2c4a3ba1588f8ee"} Aug 04 13:13:14.377876 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:14.377753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:14.398989 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:14.398932 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" podStartSLOduration=1.398914435 podStartE2EDuration="1.398914435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:13:14.395933026 +0000 UTC m=+2725.351528832" watchObservedRunningTime="2026-08-04 13:13:14.398914435 +0000 UTC m=+2725.354510228" Aug 04 13:13:25.384828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:25.384793 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:13:25.625730 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:25.625283 2578 patch_prober.go:28] interesting pod/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": read tcp 10.134.0.2:39356->10.134.0.70:15021: read: connection reset by peer" start-of-body= Aug 04 13:13:25.625730 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:25.625349 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": read tcp 10.134.0.2:39356->10.134.0.70:15021: read: connection reset by peer" Aug 04 13:13:26.457901 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:26.457867 2578 generic.go:356] "Generic (PLEG): container finished" podID="06c64759-f729-4c81-8358-5e6128959dd3" containerID="c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223" exitCode=0 Aug 04 13:13:26.458299 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:26.457944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" event={"ID":"06c64759-f729-4c81-8358-5e6128959dd3","Type":"ContainerDied","Data":"c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223"} Aug 04 13:13:26.458417 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:26.458402 2578 scope.go:117] "RemoveContainer" containerID="c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223" Aug 04 13:13:26.459518 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:26.459495 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:13:26.459577 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:26.459558 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:13:26.459613 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:26.459588 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:13:27.464956 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:27.464921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" event={"ID":"06c64759-f729-4c81-8358-5e6128959dd3","Type":"ContainerStarted","Data":"4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8"} Aug 04 13:13:28.249496 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:28.249465 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:28.249675 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:28.249647 2578 patch_prober.go:28] interesting pod/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Aug 04 13:13:28.249729 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:28.249709 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Aug 04 13:13:29.249702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:29.249658 2578 patch_prober.go:28] interesting pod/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Aug 04 13:13:29.250108 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:29.249737 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Aug 04 13:13:30.249385 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:30.249353 2578 patch_prober.go:28] interesting pod/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Aug 04 13:13:30.249598 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:30.249413 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Aug 04 13:13:31.249906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:31.249853 2578 patch_prober.go:28] interesting pod/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Aug 04 13:13:31.250406 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:31.249918 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Aug 04 13:13:32.250000 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:32.249962 2578 patch_prober.go:28] interesting pod/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Aug 04 13:13:32.250443 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:32.250044 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Aug 04 13:13:33.253587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:33.253552 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:33.253986 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:33.253948 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:33.254755 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:33.254737 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:13:36.358936 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:36.358905 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:52.259536 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.259502 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:52.308579 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.308544 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb"] Aug 04 13:13:52.308861 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.308830 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" podUID="20e6976e-1893-4ed9-8fd8-e5c47b297c81" containerName="payload-pre-processing" containerID="cri-o://be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f" gracePeriod=30 Aug 04 13:13:52.334486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.334447 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:52.404768 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.404735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:52.498725 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.498550 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:52.567291 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.567267 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:13:52.598665 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.598630 2578 generic.go:356] "Generic (PLEG): container finished" podID="20e6976e-1893-4ed9-8fd8-e5c47b297c81" containerID="be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f" exitCode=0 Aug 04 13:13:52.598851 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.598673 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" Aug 04 13:13:52.598851 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.598672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" event={"ID":"20e6976e-1893-4ed9-8fd8-e5c47b297c81","Type":"ContainerDied","Data":"be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f"} Aug 04 13:13:52.598851 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.598819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb" event={"ID":"20e6976e-1893-4ed9-8fd8-e5c47b297c81","Type":"ContainerDied","Data":"c9974f4428f64548e559a5d6b87206a489a690472600ea1c4c725f0dfd858313"} Aug 04 13:13:52.599033 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.598863 2578 scope.go:117] "RemoveContainer" containerID="be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f" Aug 04 13:13:52.609482 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.609461 2578 scope.go:117] "RemoveContainer" containerID="be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f" Aug 04 13:13:52.609744 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:13:52.609725 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f\": container with ID starting with be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f not found: ID does not exist" containerID="be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f" Aug 04 13:13:52.609818 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.609754 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f"} err="failed to get container status \"be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f\": rpc error: code = NotFound desc = could not find container \"be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f\": container with ID starting with be72f63f5074e3e90355059681bb3b492afa13685e4a6b5003233f79901a363f not found: ID does not exist" Aug 04 13:13:52.656081 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.656049 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-879bj\" (UniqueName: \"kubernetes.io/projected/20e6976e-1893-4ed9-8fd8-e5c47b297c81-kube-api-access-879bj\") pod \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " Aug 04 13:13:52.656295 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.656103 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e6976e-1893-4ed9-8fd8-e5c47b297c81-plugins-config-volume\") pod \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\" (UID: \"20e6976e-1893-4ed9-8fd8-e5c47b297c81\") " Aug 04 13:13:52.656512 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.656488 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e6976e-1893-4ed9-8fd8-e5c47b297c81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20e6976e-1893-4ed9-8fd8-e5c47b297c81" (UID: "20e6976e-1893-4ed9-8fd8-e5c47b297c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:13:52.658491 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.658466 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e6976e-1893-4ed9-8fd8-e5c47b297c81-kube-api-access-879bj" (OuterVolumeSpecName: "kube-api-access-879bj") pod "20e6976e-1893-4ed9-8fd8-e5c47b297c81" (UID: "20e6976e-1893-4ed9-8fd8-e5c47b297c81"). InnerVolumeSpecName "kube-api-access-879bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:13:52.757423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.757390 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20e6976e-1893-4ed9-8fd8-e5c47b297c81-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:13:52.757423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.757419 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879bj\" (UniqueName: \"kubernetes.io/projected/20e6976e-1893-4ed9-8fd8-e5c47b297c81-kube-api-access-879bj\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:13:52.922619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.922591 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb"] Aug 04 13:13:52.927618 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.927581 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c5286-64896bcd6d-bg9fb"] Aug 04 13:13:52.999040 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:52.999010 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:13:53.553223 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:13:53.553179 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20e6976e-1893-4ed9-8fd8-e5c47b297c81" path="/var/lib/kubelet/pods/20e6976e-1893-4ed9-8fd8-e5c47b297c81/volumes" Aug 04 13:14:00.052455 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:00.052420 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:00.152259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:00.152223 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:00.366632 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:00.366595 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9"] Aug 04 13:14:00.366909 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:00.366884 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" containerID="cri-o://95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85" gracePeriod=30 Aug 04 13:14:00.710928 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:00.710850 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:00.796543 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:00.796514 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:01.771052 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:01.771013 2578 patch_prober.go:28] interesting pod/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 13:14:01.771448 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:01.771088 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 13:14:05.209937 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.209893 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:05.610524 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.610493 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:14:05.671105 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.671071 2578 generic.go:356] "Generic (PLEG): container finished" podID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerID="95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85" exitCode=0 Aug 04 13:14:05.671318 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.671146 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" Aug 04 13:14:05.671318 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.671168 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" event={"ID":"f71e549b-d76d-4ce0-bb7c-413e7abbf819","Type":"ContainerDied","Data":"95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85"} Aug 04 13:14:05.671318 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.671231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9" event={"ID":"f71e549b-d76d-4ce0-bb7c-413e7abbf819","Type":"ContainerDied","Data":"927caabd837d38150650f75db9db835b70c8bba7b7f27e043d26f730583c0d7f"} Aug 04 13:14:05.671318 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.671258 2578 scope.go:117] "RemoveContainer" containerID="95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85" Aug 04 13:14:05.680271 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.680255 2578 scope.go:117] "RemoveContainer" containerID="195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c" Aug 04 13:14:05.688344 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.688313 2578 scope.go:117] "RemoveContainer" containerID="95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85" Aug 04 13:14:05.688613 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:14:05.688592 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85\": container with ID starting with 95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85 not found: ID does not exist" containerID="95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85" Aug 04 13:14:05.688682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.688624 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85"} err="failed to get container status \"95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85\": rpc error: code = NotFound desc = could not find container \"95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85\": container with ID starting with 95d9e449e6da149a04ce505516d971add74482e743f68c5663d5529da7c7ce85 not found: ID does not exist" Aug 04 13:14:05.688682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.688642 2578 scope.go:117] "RemoveContainer" containerID="195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c" Aug 04 13:14:05.688883 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:14:05.688866 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c\": container with ID starting with 195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c not found: ID does not exist" containerID="195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c" Aug 04 13:14:05.688926 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.688890 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c"} err="failed to get container status \"195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c\": rpc error: code = NotFound desc = could not find container \"195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c\": container with ID starting with 195694075cae3d128c2f8324e7a4e372cce92fe570477d20694e25e44de95e5c not found: ID does not exist" Aug 04 13:14:05.690173 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690144 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-envoy\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690271 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690258 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-data\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690314 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690299 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-socket\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690349 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690339 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-token\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690387 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690366 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9h8f\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-kube-api-access-v9h8f\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690439 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690396 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-credential-socket\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690439 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690429 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istiod-ca-cert\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690538 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690472 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-podinfo\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690538 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-certs\") pod \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\" (UID: \"f71e549b-d76d-4ce0-bb7c-413e7abbf819\") " Aug 04 13:14:05.690538 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690503 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-data" (OuterVolumeSpecName: "istio-data") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:05.690673 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690611 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:05.690791 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690767 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:05.691091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690983 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:14:05.691091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.690987 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.691091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.691043 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.691091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.691058 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.691091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.691009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:05.692566 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.692541 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:05.692853 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.692836 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-kube-api-access-v9h8f" (OuterVolumeSpecName: "kube-api-access-v9h8f") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "kube-api-access-v9h8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:14:05.693297 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.693260 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:14:05.693420 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.693363 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-token" (OuterVolumeSpecName: "istio-token") pod "f71e549b-d76d-4ce0-bb7c-413e7abbf819" (UID: "f71e549b-d76d-4ce0-bb7c-413e7abbf819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:14:05.792630 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.792552 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.792630 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.792578 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.792630 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.792588 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9h8f\" (UniqueName: \"kubernetes.io/projected/f71e549b-d76d-4ce0-bb7c-413e7abbf819-kube-api-access-v9h8f\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.792630 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.792597 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.792630 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.792605 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f71e549b-d76d-4ce0-bb7c-413e7abbf819-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.792630 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.792613 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f71e549b-d76d-4ce0-bb7c-413e7abbf819-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:05.999033 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:05.998992 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9"] Aug 04 13:14:06.001688 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:06.001652 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0c5286-openshift-default-6b9967d5fb-7lvd9"] Aug 04 13:14:07.553591 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:07.553555 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" path="/var/lib/kubelet/pods/f71e549b-d76d-4ce0-bb7c-413e7abbf819/volumes" Aug 04 13:14:11.255728 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.255648 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:11.400431 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.400395 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd"] Aug 04 13:14:11.400616 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.400595 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" podUID="441a6eed-f13d-43e1-b3a2-597df55a29cd" containerName="payload-pre-processing" containerID="cri-o://0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75" gracePeriod=30 Aug 04 13:14:11.444397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.444363 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:11.582674 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.582638 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:11.670555 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.670526 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:14:11.682738 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.682708 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:11.703942 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.703907 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" Aug 04 13:14:11.703942 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.703912 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" event={"ID":"441a6eed-f13d-43e1-b3a2-597df55a29cd","Type":"ContainerDied","Data":"0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75"} Aug 04 13:14:11.704220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.703950 2578 generic.go:356] "Generic (PLEG): container finished" podID="441a6eed-f13d-43e1-b3a2-597df55a29cd" containerID="0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75" exitCode=0 Aug 04 13:14:11.704220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.703987 2578 scope.go:117] "RemoveContainer" containerID="0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75" Aug 04 13:14:11.704220 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.704019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd" event={"ID":"441a6eed-f13d-43e1-b3a2-597df55a29cd","Type":"ContainerDied","Data":"61d9f82e12c50830dace2c6a5fc28d52ef52312609187f59d2c4a3ba1588f8ee"} Aug 04 13:14:11.714013 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.713995 2578 scope.go:117] "RemoveContainer" containerID="0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75" Aug 04 13:14:11.714401 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:14:11.714380 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75\": container with ID starting with 0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75 not found: ID does not exist" containerID="0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75" Aug 04 13:14:11.714494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.714407 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75"} err="failed to get container status \"0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75\": rpc error: code = NotFound desc = could not find container \"0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75\": container with ID starting with 0a04da8e6af786e3ac607c79122d4736f91515c23cc9697d33cee5a4fba5bf75 not found: ID does not exist" Aug 04 13:14:11.847682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.847652 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441a6eed-f13d-43e1-b3a2-597df55a29cd-plugins-config-volume\") pod \"441a6eed-f13d-43e1-b3a2-597df55a29cd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " Aug 04 13:14:11.847838 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.847717 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7djnw\" (UniqueName: \"kubernetes.io/projected/441a6eed-f13d-43e1-b3a2-597df55a29cd-kube-api-access-7djnw\") pod \"441a6eed-f13d-43e1-b3a2-597df55a29cd\" (UID: \"441a6eed-f13d-43e1-b3a2-597df55a29cd\") " Aug 04 13:14:11.848105 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.848079 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441a6eed-f13d-43e1-b3a2-597df55a29cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "441a6eed-f13d-43e1-b3a2-597df55a29cd" (UID: "441a6eed-f13d-43e1-b3a2-597df55a29cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:14:11.849860 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.849839 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441a6eed-f13d-43e1-b3a2-597df55a29cd-kube-api-access-7djnw" (OuterVolumeSpecName: "kube-api-access-7djnw") pod "441a6eed-f13d-43e1-b3a2-597df55a29cd" (UID: "441a6eed-f13d-43e1-b3a2-597df55a29cd"). InnerVolumeSpecName "kube-api-access-7djnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:14:11.949191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.949154 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7djnw\" (UniqueName: \"kubernetes.io/projected/441a6eed-f13d-43e1-b3a2-597df55a29cd-kube-api-access-7djnw\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:11.949191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:11.949183 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441a6eed-f13d-43e1-b3a2-597df55a29cd-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:12.036089 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:12.036057 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd"] Aug 04 13:14:12.043392 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:12.043362 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c1a14d-6f99d7576-2frqd"] Aug 04 13:14:12.110790 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:12.110694 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:13.553282 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:13.553250 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441a6eed-f13d-43e1-b3a2-597df55a29cd" path="/var/lib/kubelet/pods/441a6eed-f13d-43e1-b3a2-597df55a29cd/volumes" Aug 04 13:14:19.485779 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:19.485721 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:19.552901 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:19.552865 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:19.761295 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:19.761209 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s"] Aug 04 13:14:19.761508 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:19.761462 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" containerID="cri-o://4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8" gracePeriod=30 Aug 04 13:14:19.770782 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:19.770761 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:20.090441 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.090403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:20.371179 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.371080 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:20.694672 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.694584 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs"] Aug 04 13:14:20.695061 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695030 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20e6976e-1893-4ed9-8fd8-e5c47b297c81" containerName="payload-pre-processing" Aug 04 13:14:20.695061 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695046 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="20e6976e-1893-4ed9-8fd8-e5c47b297c81" containerName="payload-pre-processing" Aug 04 13:14:20.695061 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695056 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" Aug 04 13:14:20.695061 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695061 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695076 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441a6eed-f13d-43e1-b3a2-597df55a29cd" containerName="payload-pre-processing" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695082 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="441a6eed-f13d-43e1-b3a2-597df55a29cd" containerName="payload-pre-processing" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695094 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695102 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695203 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695211 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71e549b-d76d-4ce0-bb7c-413e7abbf819" containerName="istio-proxy" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695219 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="441a6eed-f13d-43e1-b3a2-597df55a29cd" containerName="payload-pre-processing" Aug 04 13:14:20.695407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.695226 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="20e6976e-1893-4ed9-8fd8-e5c47b297c81" containerName="payload-pre-processing" Aug 04 13:14:20.699937 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.699922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.704002 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.703980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6a00c1-openshift-default-dockercfg-7b8sv\"" Aug 04 13:14:20.721722 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.721680 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs"] Aug 04 13:14:20.727672 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.727646 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:20.835096 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b2e22-81bf-4189-9af0-0abed7af5230-istiod-ca-cert\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-podinfo\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-envoy\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-token\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77dlq\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-kube-api-access-77dlq\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835286 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-data\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835725 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-credential-socket\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835725 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-socket\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.835725 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.835420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-certs\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.935978 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.935936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b2e22-81bf-4189-9af0-0abed7af5230-istiod-ca-cert\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.935978 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.935987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-podinfo\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-envoy\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-token\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77dlq\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-kube-api-access-77dlq\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-data\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936402 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-credential-socket\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936402 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-socket\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936402 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-certs\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936708 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936675 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-credential-socket\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936812 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-data\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936812 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-certs\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936901 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b2e22-81bf-4189-9af0-0abed7af5230-istiod-ca-cert\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.936901 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.936833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-socket\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.939153 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.939104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-envoy\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.939353 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.939336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-podinfo\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.946700 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.946639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-token\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:20.946833 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:20.946815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77dlq\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-kube-api-access-77dlq\") pod \"e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:21.010655 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.010605 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:21.160808 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.160780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs"] Aug 04 13:14:21.162763 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:14:21.162734 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2b2e22_81bf_4189_9af0_0abed7af5230.slice/crio-cd2bd103748ce01a268a65243218a35cda41d071acd9b3360dbaf6991f4d784a WatchSource:0}: Error finding container cd2bd103748ce01a268a65243218a35cda41d071acd9b3360dbaf6991f4d784a: Status 404 returned error can't find the container with id cd2bd103748ce01a268a65243218a35cda41d071acd9b3360dbaf6991f4d784a Aug 04 13:14:21.165034 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.165002 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:21.165163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.165096 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:21.165163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.165146 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:21.192290 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.192244 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:21.761345 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.761310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" event={"ID":"cf2b2e22-81bf-4189-9af0-0abed7af5230","Type":"ContainerStarted","Data":"1b08fa688264bfbb7883e025953a5c301741c5dbc3cbfddae0e40e43f43ebcd4"} Aug 04 13:14:21.761750 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.761373 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" event={"ID":"cf2b2e22-81bf-4189-9af0-0abed7af5230","Type":"ContainerStarted","Data":"cd2bd103748ce01a268a65243218a35cda41d071acd9b3360dbaf6991f4d784a"} Aug 04 13:14:21.794247 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:21.794195 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podStartSLOduration=1.794173612 podStartE2EDuration="1.794173612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:14:21.793554508 +0000 UTC m=+2792.749150337" watchObservedRunningTime="2026-08-04 13:14:21.794173612 +0000 UTC m=+2792.749769403" Aug 04 13:14:22.013429 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:22.013335 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:22.019670 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:22.019637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:22.767089 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:22.767049 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:22.768023 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:22.768002 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:23.321979 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:23.321943 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:23.421398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:23.421362 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:23.784619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:23.784587 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:23.912953 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:23.912916 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:24.903148 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:24.903086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:25.021054 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.021025 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:14:25.076806 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076728 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzq69\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-kube-api-access-mzq69\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.076806 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076789 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-envoy\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076813 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c64759-f729-4c81-8358-5e6128959dd3-istio-podinfo\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076841 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c64759-f729-4c81-8358-5e6128959dd3-istiod-ca-cert\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076871 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-socket\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076903 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-certs\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076934 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-istio-token\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.076982 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-credential-socket\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077015 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-data\") pod \"06c64759-f729-4c81-8358-5e6128959dd3\" (UID: \"06c64759-f729-4c81-8358-5e6128959dd3\") " Aug 04 13:14:25.077475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077256 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c64759-f729-4c81-8358-5e6128959dd3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:14:25.077475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077446 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-data" (OuterVolumeSpecName: "istio-data") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:25.077475 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077457 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c64759-f729-4c81-8358-5e6128959dd3-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.077798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077651 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:25.077798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077780 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:25.077945 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.077915 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:25.079315 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.079289 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06c64759-f729-4c81-8358-5e6128959dd3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:14:25.079465 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.079444 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-kube-api-access-mzq69" (OuterVolumeSpecName: "kube-api-access-mzq69") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "kube-api-access-mzq69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:14:25.079725 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.079697 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-istio-token" (OuterVolumeSpecName: "istio-token") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:14:25.079827 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.079723 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06c64759-f729-4c81-8358-5e6128959dd3" (UID: "06c64759-f729-4c81-8358-5e6128959dd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:14:25.178175 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178114 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzq69\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-kube-api-access-mzq69\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178175 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178172 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178175 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178184 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c64759-f729-4c81-8358-5e6128959dd3-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178413 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178192 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178413 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178200 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178413 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178208 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c64759-f729-4c81-8358-5e6128959dd3-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178413 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178215 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.178413 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.178223 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c64759-f729-4c81-8358-5e6128959dd3-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:14:25.233719 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.233687 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:25.561969 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.561939 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:25.752746 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.752713 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft"] Aug 04 13:14:25.753218 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.753194 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" Aug 04 13:14:25.753218 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.753217 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" Aug 04 13:14:25.753400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.753249 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" Aug 04 13:14:25.753400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.753259 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" Aug 04 13:14:25.753400 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.753362 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" Aug 04 13:14:25.756777 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.756759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.759573 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.759526 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-6a00c1\"" Aug 04 13:14:25.759809 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.759781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-6a00c1-dockercfg-77988\"" Aug 04 13:14:25.766344 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.766318 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft"] Aug 04 13:14:25.783035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.783002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75pw\" (UniqueName: \"kubernetes.io/projected/80fedacc-35c3-4873-a2c8-151e6804b11b-kube-api-access-w75pw\") pod \"payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.783228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.783074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fedacc-35c3-4873-a2c8-151e6804b11b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.785902 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.785875 2578 generic.go:356] "Generic (PLEG): container finished" podID="06c64759-f729-4c81-8358-5e6128959dd3" containerID="4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8" exitCode=0 Aug 04 13:14:25.786023 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.785932 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" Aug 04 13:14:25.786023 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.785956 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" event={"ID":"06c64759-f729-4c81-8358-5e6128959dd3","Type":"ContainerDied","Data":"4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8"} Aug 04 13:14:25.786023 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.786011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s" event={"ID":"06c64759-f729-4c81-8358-5e6128959dd3","Type":"ContainerDied","Data":"8e1289564f5c5817a6f9a7716032078a6a49272f973ce12636cd873d35e2889c"} Aug 04 13:14:25.786248 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.786033 2578 scope.go:117] "RemoveContainer" containerID="4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8" Aug 04 13:14:25.795423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.795407 2578 scope.go:117] "RemoveContainer" containerID="c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223" Aug 04 13:14:25.804181 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.803880 2578 scope.go:117] "RemoveContainer" containerID="4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8" Aug 04 13:14:25.804296 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:14:25.804212 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8\": container with ID starting with 4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8 not found: ID does not exist" containerID="4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8" Aug 04 13:14:25.804296 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.804241 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8"} err="failed to get container status \"4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8\": rpc error: code = NotFound desc = could not find container \"4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8\": container with ID starting with 4a53c30c434cdbda9750c59b14237fc1c163bac22c05f1958f132f9eb86058a8 not found: ID does not exist" Aug 04 13:14:25.804296 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.804257 2578 scope.go:117] "RemoveContainer" containerID="c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223" Aug 04 13:14:25.804527 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:14:25.804497 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223\": container with ID starting with c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223 not found: ID does not exist" containerID="c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223" Aug 04 13:14:25.804590 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.804532 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223"} err="failed to get container status \"c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223\": rpc error: code = NotFound desc = could not find container \"c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223\": container with ID starting with c7e1cdfb1066eb3205214b4e15abc948641b6000cc2b32c68d544e6034944223 not found: ID does not exist" Aug 04 13:14:25.806004 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.805982 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s"] Aug 04 13:14:25.810516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.810495 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c1a14d-openshift-default-68557657fd-6579s"] Aug 04 13:14:25.884526 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.884489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fedacc-35c3-4873-a2c8-151e6804b11b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.884718 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.884619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75pw\" (UniqueName: \"kubernetes.io/projected/80fedacc-35c3-4873-a2c8-151e6804b11b-kube-api-access-w75pw\") pod \"payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.885259 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.885243 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fedacc-35c3-4873-a2c8-151e6804b11b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.897271 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.897248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75pw\" (UniqueName: \"kubernetes.io/projected/80fedacc-35c3-4873-a2c8-151e6804b11b-kube-api-access-w75pw\") pod \"payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:25.933370 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:25.933340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:26.069993 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:26.069961 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:26.215601 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:26.215576 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft"] Aug 04 13:14:26.217964 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:14:26.217930 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fedacc_35c3_4873_a2c8_151e6804b11b.slice/crio-3ffdc65ef94251f8e6232a65cd99bb84a5e3efd87284100132effc99962f552d WatchSource:0}: Error finding container 3ffdc65ef94251f8e6232a65cd99bb84a5e3efd87284100132effc99962f552d: Status 404 returned error can't find the container with id 3ffdc65ef94251f8e6232a65cd99bb84a5e3efd87284100132effc99962f552d Aug 04 13:14:26.793622 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:26.793583 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" event={"ID":"80fedacc-35c3-4873-a2c8-151e6804b11b","Type":"ContainerStarted","Data":"083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e"} Aug 04 13:14:26.793811 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:26.793635 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" event={"ID":"80fedacc-35c3-4873-a2c8-151e6804b11b","Type":"ContainerStarted","Data":"3ffdc65ef94251f8e6232a65cd99bb84a5e3efd87284100132effc99962f552d"} Aug 04 13:14:26.793811 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:26.793666 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:26.813581 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:26.813527 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" podStartSLOduration=1.813510948 podStartE2EDuration="1.813510948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:14:26.811160191 +0000 UTC m=+2797.766756009" watchObservedRunningTime="2026-08-04 13:14:26.813510948 +0000 UTC m=+2797.769106741" Aug 04 13:14:27.553192 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:27.553158 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c64759-f729-4c81-8358-5e6128959dd3" path="/var/lib/kubelet/pods/06c64759-f729-4c81-8358-5e6128959dd3/volumes" Aug 04 13:14:37.768719 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.768683 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:37.769188 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.768750 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:37.800809 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.800779 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:14:37.861395 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.861362 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerID="1b08fa688264bfbb7883e025953a5c301741c5dbc3cbfddae0e40e43f43ebcd4" exitCode=0 Aug 04 13:14:37.861562 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.861447 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" event={"ID":"cf2b2e22-81bf-4189-9af0-0abed7af5230","Type":"ContainerDied","Data":"1b08fa688264bfbb7883e025953a5c301741c5dbc3cbfddae0e40e43f43ebcd4"} Aug 04 13:14:37.861923 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.861910 2578 scope.go:117] "RemoveContainer" containerID="1b08fa688264bfbb7883e025953a5c301741c5dbc3cbfddae0e40e43f43ebcd4" Aug 04 13:14:37.863138 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.863094 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:37.863229 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.863190 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:37.863229 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:37.863224 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:38.868840 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:38.868808 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" event={"ID":"cf2b2e22-81bf-4189-9af0-0abed7af5230","Type":"ContainerStarted","Data":"288fbb3d33d67f68613cb923ad09d99e642752d9b77499efe62e61ab9b133a45"} Aug 04 13:14:39.011874 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:39.011824 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:39.012218 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:39.012179 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:39.012461 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:39.012435 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:39.488434 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:39.488401 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:40.011873 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:40.011834 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:40.012494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:40.012460 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:41.011619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:41.011578 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:41.011913 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:41.011881 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:42.011360 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:42.011312 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:42.011584 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:42.011401 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:43.011688 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:43.011646 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:43.012313 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:43.012276 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:44.011789 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:44.011750 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Aug 04 13:14:44.012190 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:44.011805 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Aug 04 13:14:45.016134 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:45.016091 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:45.016562 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:45.016364 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:45.016938 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:45.016920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:14:48.785011 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:48.784966 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:49.107769 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.107738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:49.388913 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.388840 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs"] Aug 04 13:14:49.389322 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.389303 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c64759-f729-4c81-8358-5e6128959dd3" containerName="istio-proxy" Aug 04 13:14:49.391402 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.391387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.394424 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.394400 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c1c995-openshift-default-dockercfg-xrdkh\"" Aug 04 13:14:49.404234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.404210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs"] Aug 04 13:14:49.530354 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istiod-ca-cert\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530362 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-data\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-credential-socket\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vfj\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-kube-api-access-d9vfj\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-envoy\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530549 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-socket\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530779 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-podinfo\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530779 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-certs\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.530779 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.530743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-token\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.631728 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-certs\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.631728 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-token\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istiod-ca-cert\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-data\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-credential-socket\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vfj\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-kube-api-access-d9vfj\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632020 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.631990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-envoy\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-socket\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-podinfo\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-certs\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632190 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-data\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632397 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632353 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-credential-socket\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632660 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632535 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-socket\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.632660 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.632613 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istiod-ca-cert\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.634766 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.634727 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-envoy\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.634966 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.634946 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-podinfo\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.643280 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.643226 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-token\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.643399 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.643331 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vfj\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-kube-api-access-d9vfj\") pod \"e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.672805 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.672773 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:49.702826 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.702794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:49.741858 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.741825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:49.843378 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.843350 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs"] Aug 04 13:14:49.845109 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:14:49.845082 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b00cdcc_d8d5_43cc_86e0_1c243eb411fa.slice/crio-33e29d32f87febdcbf1186dfdc8fa5d1dd2c1f48be3e6d3d3b6be13eef0902c8 WatchSource:0}: Error finding container 33e29d32f87febdcbf1186dfdc8fa5d1dd2c1f48be3e6d3d3b6be13eef0902c8: Status 404 returned error can't find the container with id 33e29d32f87febdcbf1186dfdc8fa5d1dd2c1f48be3e6d3d3b6be13eef0902c8 Aug 04 13:14:49.847477 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.847443 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:49.847578 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.847508 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:49.847578 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.847535 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:14:49.925955 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.925918 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" event={"ID":"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa","Type":"ContainerStarted","Data":"70743cf614dcedfe6f9a7e7a1d770d54a8c035879b28209e23186a86376fc118"} Aug 04 13:14:49.925955 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.925976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" event={"ID":"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa","Type":"ContainerStarted","Data":"33e29d32f87febdcbf1186dfdc8fa5d1dd2c1f48be3e6d3d3b6be13eef0902c8"} Aug 04 13:14:49.952041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:49.951981 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podStartSLOduration=0.951964932 podStartE2EDuration="951.964932ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:14:49.949547797 +0000 UTC m=+2820.905143601" watchObservedRunningTime="2026-08-04 13:14:49.951964932 +0000 UTC m=+2820.907560787" Aug 04 13:14:50.703110 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:50.703076 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:50.708261 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:50.708233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:50.931096 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:50.931067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:50.932565 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:50.932545 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:14:52.001188 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:52.001138 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:52.101722 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:52.101693 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:53.165933 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.165905 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:53.437766 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.437660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:53.828614 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.828488 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:53.896521 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.896486 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g"] Aug 04 13:14:53.898920 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.898895 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:53.899350 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.899333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:53.902102 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.902070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-c1c995\"" Aug 04 13:14:53.911201 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.911175 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g"] Aug 04 13:14:53.976790 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.976752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04e3ba8-7868-405c-b444-5e8951fee570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:53.976968 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:53.976882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv9n\" (UniqueName: \"kubernetes.io/projected/d04e3ba8-7868-405c-b444-5e8951fee570-kube-api-access-btv9n\") pod \"payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.077872 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.077833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv9n\" (UniqueName: \"kubernetes.io/projected/d04e3ba8-7868-405c-b444-5e8951fee570-kube-api-access-btv9n\") pod \"payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.078049 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.077916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04e3ba8-7868-405c-b444-5e8951fee570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.078941 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.078891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04e3ba8-7868-405c-b444-5e8951fee570-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.087731 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.087704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv9n\" (UniqueName: \"kubernetes.io/projected/d04e3ba8-7868-405c-b444-5e8951fee570-kube-api-access-btv9n\") pod \"payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.211234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.211196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.288894 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.288863 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:14:54.561168 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.561143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g"] Aug 04 13:14:54.563224 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:14:54.563180 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04e3ba8_7868_405c_b444_5e8951fee570.slice/crio-6a338ba671956cebd8271a2b22cefb1ae459461792a4aed7124bcf8c630b0118 WatchSource:0}: Error finding container 6a338ba671956cebd8271a2b22cefb1ae459461792a4aed7124bcf8c630b0118: Status 404 returned error can't find the container with id 6a338ba671956cebd8271a2b22cefb1ae459461792a4aed7124bcf8c630b0118 Aug 04 13:14:54.954427 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.954390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" event={"ID":"d04e3ba8-7868-405c-b444-5e8951fee570","Type":"ContainerStarted","Data":"d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f"} Aug 04 13:14:54.954595 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.954445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" event={"ID":"d04e3ba8-7868-405c-b444-5e8951fee570","Type":"ContainerStarted","Data":"6a338ba671956cebd8271a2b22cefb1ae459461792a4aed7124bcf8c630b0118"} Aug 04 13:14:54.954595 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.954527 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:14:54.974113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:14:54.974069 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" podStartSLOduration=1.974053075 podStartE2EDuration="1.974053075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:14:54.972091577 +0000 UTC m=+2825.927687396" watchObservedRunningTime="2026-08-04 13:14:54.974053075 +0000 UTC m=+2825.929648870" Aug 04 13:15:05.932722 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:05.932651 2578 patch_prober.go:28] interesting pod/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Aug 04 13:15:05.933090 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:05.932714 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Aug 04 13:15:05.961531 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:05.961503 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:15:06.012993 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:06.012907 2578 generic.go:356] "Generic (PLEG): container finished" podID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerID="70743cf614dcedfe6f9a7e7a1d770d54a8c035879b28209e23186a86376fc118" exitCode=0 Aug 04 13:15:06.013185 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:06.012988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" event={"ID":"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa","Type":"ContainerDied","Data":"70743cf614dcedfe6f9a7e7a1d770d54a8c035879b28209e23186a86376fc118"} Aug 04 13:15:06.013487 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:06.013472 2578 scope.go:117] "RemoveContainer" containerID="70743cf614dcedfe6f9a7e7a1d770d54a8c035879b28209e23186a86376fc118" Aug 04 13:15:06.014714 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:06.014650 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:15:06.014828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:06.014734 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:15:06.014828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:06.014786 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:15:07.023320 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:07.023284 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" event={"ID":"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa","Type":"ContainerStarted","Data":"9c2442bcbbe4fd1f26f17285e1146294a9f9fd0882d44e2e86bfb728bc66223a"} Aug 04 13:15:07.704062 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:07.704018 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:15:07.704522 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:07.704492 2578 patch_prober.go:28] interesting pod/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Aug 04 13:15:07.704618 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:07.704552 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Aug 04 13:15:08.704289 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:08.704241 2578 patch_prober.go:28] interesting pod/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Aug 04 13:15:08.704797 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:08.704316 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Aug 04 13:15:09.704161 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:09.704096 2578 patch_prober.go:28] interesting pod/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Aug 04 13:15:09.704478 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:09.704448 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Aug 04 13:15:10.703413 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:10.703374 2578 patch_prober.go:28] interesting pod/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Aug 04 13:15:10.703633 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:10.703443 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Aug 04 13:15:11.704036 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:11.703991 2578 patch_prober.go:28] interesting pod/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Aug 04 13:15:11.704541 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:11.704062 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Aug 04 13:15:12.707554 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:12.707522 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:15:12.707913 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:12.707885 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:15:12.708646 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:12.708624 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:15:16.922676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:16.922637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:15:25.886349 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:25.886314 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:15:58.303168 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.303111 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg"] Aug 04 13:15:58.307803 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.307781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.312239 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.312214 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c1c995\"/\"auth-test-model-c1c995-kserve-self-signed-certs\"" Aug 04 13:15:58.312386 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.312215 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c1c995\"/\"kube-root-ca.crt\"" Aug 04 13:15:58.312386 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.312220 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c1c995\"/\"default-dockercfg-gltvg\"" Aug 04 13:15:58.312386 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.312241 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c1c995\"/\"openshift-service-ca.crt\"" Aug 04 13:15:58.316188 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.315857 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg"] Aug 04 13:15:58.332886 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.332853 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:15:58.430500 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.430461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-dshm\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.430710 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.430519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-tmp-dir\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.430710 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.430546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-home\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.430710 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.430570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2786cbe9-d9ac-4108-b645-c5174fced85b-tls-certs\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.430710 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.430630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc65t\" (UniqueName: \"kubernetes.io/projected/2786cbe9-d9ac-4108-b645-c5174fced85b-kube-api-access-gc65t\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.430710 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.430702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-model-cache\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532042 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-dshm\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532245 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-tmp-dir\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532245 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-home\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532245 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2786cbe9-d9ac-4108-b645-c5174fced85b-tls-certs\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532245 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc65t\" (UniqueName: \"kubernetes.io/projected/2786cbe9-d9ac-4108-b645-c5174fced85b-kube-api-access-gc65t\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532245 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-model-cache\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532529 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532503 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-tmp-dir\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532592 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532571 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-home\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.532637 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.532619 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-model-cache\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.534476 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.534457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-dshm\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.534800 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.534781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2786cbe9-d9ac-4108-b645-c5174fced85b-tls-certs\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.544818 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.544794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc65t\" (UniqueName: \"kubernetes.io/projected/2786cbe9-d9ac-4108-b645-c5174fced85b-kube-api-access-gc65t\") pod \"auth-test-model-c1c995-kserve-7554c4fbf9-df5bg\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.619732 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.619689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:15:58.770449 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:58.770424 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg"] Aug 04 13:15:58.772275 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:15:58.772241 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2786cbe9_d9ac_4108_b645_c5174fced85b.slice/crio-401e2c7141f7ff343ffe46c40a7c0d9bd665ad867dca2ce73176dedfaaac7248 WatchSource:0}: Error finding container 401e2c7141f7ff343ffe46c40a7c0d9bd665ad867dca2ce73176dedfaaac7248: Status 404 returned error can't find the container with id 401e2c7141f7ff343ffe46c40a7c0d9bd665ad867dca2ce73176dedfaaac7248 Aug 04 13:15:59.303948 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:59.303916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" event={"ID":"2786cbe9-d9ac-4108-b645-c5174fced85b","Type":"ContainerStarted","Data":"4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e"} Aug 04 13:15:59.303948 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:59.303957 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" event={"ID":"2786cbe9-d9ac-4108-b645-c5174fced85b","Type":"ContainerStarted","Data":"401e2c7141f7ff343ffe46c40a7c0d9bd665ad867dca2ce73176dedfaaac7248"} Aug 04 13:15:59.324602 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:15:59.324550 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" podStartSLOduration=1.324533267 podStartE2EDuration="1.324533267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:15:59.323723993 +0000 UTC m=+2890.279319786" watchObservedRunningTime="2026-08-04 13:15:59.324533267 +0000 UTC m=+2890.280129060" Aug 04 13:16:08.620313 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:16:08.620273 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:16:08.620718 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:16:08.620333 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:16:08.633339 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:16:08.633304 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:16:09.368138 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:16:09.368101 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:17:49.687477 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:17:49.687447 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:17:49.696637 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:17:49.696617 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:17:55.172486 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:17:55.172448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:17:55.229418 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:17:55.229380 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:18:42.776633 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:18:42.776550 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:18:54.430488 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:18:54.430450 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:18:55.328836 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:18:55.328803 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg"] Aug 04 13:18:55.355328 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:18:55.355299 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:19:09.631678 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:09.631645 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:19:10.349480 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.349442 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" podUID="2786cbe9-d9ac-4108-b645-c5174fced85b" containerName="main" containerID="cri-o://4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e" gracePeriod=45 Aug 04 13:19:10.612370 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.612307 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:19:10.760845 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.760788 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2786cbe9-d9ac-4108-b645-c5174fced85b-tls-certs\") pod \"2786cbe9-d9ac-4108-b645-c5174fced85b\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.760861 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-dshm\") pod \"2786cbe9-d9ac-4108-b645-c5174fced85b\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.760950 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-tmp-dir\") pod \"2786cbe9-d9ac-4108-b645-c5174fced85b\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.760975 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-model-cache\") pod \"2786cbe9-d9ac-4108-b645-c5174fced85b\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761003 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-home\") pod \"2786cbe9-d9ac-4108-b645-c5174fced85b\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761052 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc65t\" (UniqueName: \"kubernetes.io/projected/2786cbe9-d9ac-4108-b645-c5174fced85b-kube-api-access-gc65t\") pod \"2786cbe9-d9ac-4108-b645-c5174fced85b\" (UID: \"2786cbe9-d9ac-4108-b645-c5174fced85b\") " Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761265 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2786cbe9-d9ac-4108-b645-c5174fced85b" (UID: "2786cbe9-d9ac-4108-b645-c5174fced85b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761280 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2786cbe9-d9ac-4108-b645-c5174fced85b" (UID: "2786cbe9-d9ac-4108-b645-c5174fced85b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:19:10.761368 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761342 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-home" (OuterVolumeSpecName: "home") pod "2786cbe9-d9ac-4108-b645-c5174fced85b" (UID: "2786cbe9-d9ac-4108-b645-c5174fced85b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:19:10.761705 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761495 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-tmp-dir\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:19:10.761705 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761509 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-model-cache\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:19:10.761705 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.761518 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-home\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:19:10.763361 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.763315 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-dshm" (OuterVolumeSpecName: "dshm") pod "2786cbe9-d9ac-4108-b645-c5174fced85b" (UID: "2786cbe9-d9ac-4108-b645-c5174fced85b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:19:10.763473 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.763394 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2786cbe9-d9ac-4108-b645-c5174fced85b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2786cbe9-d9ac-4108-b645-c5174fced85b" (UID: "2786cbe9-d9ac-4108-b645-c5174fced85b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 13:19:10.763473 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.763408 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2786cbe9-d9ac-4108-b645-c5174fced85b-kube-api-access-gc65t" (OuterVolumeSpecName: "kube-api-access-gc65t") pod "2786cbe9-d9ac-4108-b645-c5174fced85b" (UID: "2786cbe9-d9ac-4108-b645-c5174fced85b"). InnerVolumeSpecName "kube-api-access-gc65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:19:10.862701 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.862619 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc65t\" (UniqueName: \"kubernetes.io/projected/2786cbe9-d9ac-4108-b645-c5174fced85b-kube-api-access-gc65t\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:19:10.862701 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.862649 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2786cbe9-d9ac-4108-b645-c5174fced85b-tls-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:19:10.862701 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:10.862659 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2786cbe9-d9ac-4108-b645-c5174fced85b-dshm\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:19:11.304634 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.304549 2578 generic.go:356] "Generic (PLEG): container finished" podID="2786cbe9-d9ac-4108-b645-c5174fced85b" containerID="4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e" exitCode=0 Aug 04 13:19:11.304634 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.304591 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" Aug 04 13:19:11.304880 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.304643 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" event={"ID":"2786cbe9-d9ac-4108-b645-c5174fced85b","Type":"ContainerDied","Data":"4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e"} Aug 04 13:19:11.304880 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.304692 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg" event={"ID":"2786cbe9-d9ac-4108-b645-c5174fced85b","Type":"ContainerDied","Data":"401e2c7141f7ff343ffe46c40a7c0d9bd665ad867dca2ce73176dedfaaac7248"} Aug 04 13:19:11.304880 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.304715 2578 scope.go:117] "RemoveContainer" containerID="4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e" Aug 04 13:19:11.314479 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.314463 2578 scope.go:117] "RemoveContainer" containerID="4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e" Aug 04 13:19:11.314760 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:19:11.314737 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e\": container with ID starting with 4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e not found: ID does not exist" containerID="4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e" Aug 04 13:19:11.314821 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.314770 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e"} err="failed to get container status \"4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e\": rpc error: code = NotFound desc = could not find container \"4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e\": container with ID starting with 4ddf9da8a580d3e6cbfdf8eb06f71a7a8b0ee633dc9addac29425531776d270e not found: ID does not exist" Aug 04 13:19:11.329326 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.329306 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg"] Aug 04 13:19:11.334131 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.334096 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c1c995/auth-test-model-c1c995-kserve-7554c4fbf9-df5bg"] Aug 04 13:19:11.553163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:11.553113 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2786cbe9-d9ac-4108-b645-c5174fced85b" path="/var/lib/kubelet/pods/2786cbe9-d9ac-4108-b645-c5174fced85b/volumes" Aug 04 13:19:15.338393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:15.338362 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:19:41.606646 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:19:41.606608 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:21:06.525967 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:21:06.525929 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:21:07.543827 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:21:07.543788 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:21:29.476593 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:21:29.476559 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:21:34.264924 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:21:34.264884 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:22:06.377253 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:22:06.377191 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:22:36.036478 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:22:36.036444 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:22:49.730935 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:22:49.730905 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:22:49.741769 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:22:49.741745 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:23:15.535408 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:15.535318 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:16.504462 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:16.504424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:36.693802 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:36.693765 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:37.027039 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:37.026952 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:37.242376 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:37.242332 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:39.928985 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:39.928950 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:40.279481 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:40.279403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:41.067300 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.067257 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:41.277676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.277639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:41.584337 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.584301 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:41.685528 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.685497 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:23:41.703702 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.703666 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb"] Aug 04 13:23:41.704110 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.704097 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2786cbe9-d9ac-4108-b645-c5174fced85b" containerName="main" Aug 04 13:23:41.704188 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.704111 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2786cbe9-d9ac-4108-b645-c5174fced85b" containerName="main" Aug 04 13:23:41.704225 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.704214 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2786cbe9-d9ac-4108-b645-c5174fced85b" containerName="main" Aug 04 13:23:41.707512 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.707489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:41.710852 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.710829 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2ef922\"" Aug 04 13:23:41.719785 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.719762 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb"] Aug 04 13:23:41.732448 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.732410 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h"] Aug 04 13:23:41.736005 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.735983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.738986 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.738966 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2ef922-dockercfg-vczxr\"" Aug 04 13:23:41.746730 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.746706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h"] Aug 04 13:23:41.858332 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.858302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664be107-c4e9-4cb5-9d94-522760c54663-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.858492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.858343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74tp\" (UniqueName: \"kubernetes.io/projected/664be107-c4e9-4cb5-9d94-522760c54663-kube-api-access-p74tp\") pod \"payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.858492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.858415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:41.858572 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.858489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kcs\" (UniqueName: \"kubernetes.io/projected/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-kube-api-access-p8kcs\") pod \"payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:41.959349 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.959309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:41.959559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.959374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kcs\" (UniqueName: \"kubernetes.io/projected/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-kube-api-access-p8kcs\") pod \"payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:41.959559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.959465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664be107-c4e9-4cb5-9d94-522760c54663-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.959559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.959497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74tp\" (UniqueName: \"kubernetes.io/projected/664be107-c4e9-4cb5-9d94-522760c54663-kube-api-access-p74tp\") pod \"payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.960164 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.960111 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:41.960327 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.960307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664be107-c4e9-4cb5-9d94-522760c54663-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.972337 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.972312 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74tp\" (UniqueName: \"kubernetes.io/projected/664be107-c4e9-4cb5-9d94-522760c54663-kube-api-access-p74tp\") pod \"payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:41.972337 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:41.972326 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kcs\" (UniqueName: \"kubernetes.io/projected/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-kube-api-access-p8kcs\") pod \"payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:42.019242 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.019214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:42.049519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.049486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:42.180639 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.180598 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb"] Aug 04 13:23:42.182627 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:23:42.182599 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea01e1c_2ded_440e_bc5a_7f22ccc1e7a5.slice/crio-34d2c28864625691865c76962acdf0fcb70d6a0da6485749a0d130f32aaa9394 WatchSource:0}: Error finding container 34d2c28864625691865c76962acdf0fcb70d6a0da6485749a0d130f32aaa9394: Status 404 returned error can't find the container with id 34d2c28864625691865c76962acdf0fcb70d6a0da6485749a0d130f32aaa9394 Aug 04 13:23:42.212847 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.212822 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h"] Aug 04 13:23:42.215589 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:23:42.215558 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664be107_c4e9_4cb5_9d94_522760c54663.slice/crio-0526c72f6c7305009bf26a98c13c115e20dcff915281e8220a10be331b99f3d5 WatchSource:0}: Error finding container 0526c72f6c7305009bf26a98c13c115e20dcff915281e8220a10be331b99f3d5: Status 404 returned error can't find the container with id 0526c72f6c7305009bf26a98c13c115e20dcff915281e8220a10be331b99f3d5 Aug 04 13:23:42.754839 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.754802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" event={"ID":"664be107-c4e9-4cb5-9d94-522760c54663","Type":"ContainerStarted","Data":"3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee"} Aug 04 13:23:42.755048 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.754950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:23:42.755048 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.755015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" event={"ID":"664be107-c4e9-4cb5-9d94-522760c54663","Type":"ContainerStarted","Data":"0526c72f6c7305009bf26a98c13c115e20dcff915281e8220a10be331b99f3d5"} Aug 04 13:23:42.756416 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.756392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" event={"ID":"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5","Type":"ContainerStarted","Data":"bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99"} Aug 04 13:23:42.756525 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.756428 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" event={"ID":"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5","Type":"ContainerStarted","Data":"34d2c28864625691865c76962acdf0fcb70d6a0da6485749a0d130f32aaa9394"} Aug 04 13:23:42.756525 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.756511 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:42.775968 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.775923 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" podStartSLOduration=1.775908805 podStartE2EDuration="1.775908805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:23:42.772741091 +0000 UTC m=+3353.728336884" watchObservedRunningTime="2026-08-04 13:23:42.775908805 +0000 UTC m=+3353.731504611" Aug 04 13:23:42.791321 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:42.791273 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" podStartSLOduration=1.791258586 podStartE2EDuration="1.791258586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:23:42.789277245 +0000 UTC m=+3353.744873040" watchObservedRunningTime="2026-08-04 13:23:42.791258586 +0000 UTC m=+3353.746854380" Aug 04 13:23:53.764254 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:53.764216 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:23:53.764647 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:23:53.764445 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:24:04.948823 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:04.948787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:05.931698 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:05.931659 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn"] Aug 04 13:24:05.935387 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:05.935365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:05.938690 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:05.938671 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-77ce5c-openshift-default-dockercfg-bwfsz\"" Aug 04 13:24:05.948150 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:05.948106 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn"] Aug 04 13:24:06.007045 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.006990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-token\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxj24\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-kube-api-access-qxj24\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-podinfo\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-data\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-credential-socket\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-certs\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-envoy\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istiod-ca-cert\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.007533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.007376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-socket\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.108887 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.108852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-envoy\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.108887 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.108892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istiod-ca-cert\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.108917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-socket\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.108977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-token\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxj24\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-kube-api-access-qxj24\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-podinfo\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-data\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-credential-socket\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109191 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-certs\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109573 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-socket\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109573 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109542 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-certs\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109689 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-data\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109795 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109774 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istiod-ca-cert\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.109855 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.109837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-credential-socket\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.111748 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.111728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-envoy\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.111844 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.111792 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-podinfo\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.121729 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.121706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-token\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.122326 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.122303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxj24\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-kube-api-access-qxj24\") pod \"e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.247453 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.247368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:06.395764 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.395733 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn"] Aug 04 13:24:06.397167 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:24:06.397137 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45c985e_6f19_41ba_9744_07a1c5a7c20f.slice/crio-db35f26149079c58712ed04f78a447fc83ac5a50b798fd13ca3e7c6621ef496f WatchSource:0}: Error finding container db35f26149079c58712ed04f78a447fc83ac5a50b798fd13ca3e7c6621ef496f: Status 404 returned error can't find the container with id db35f26149079c58712ed04f78a447fc83ac5a50b798fd13ca3e7c6621ef496f Aug 04 13:24:06.399338 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.399307 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:24:06.399429 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.399371 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:24:06.399429 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.399404 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:24:06.625890 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.625858 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:06.887524 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.887425 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" event={"ID":"d45c985e-6f19-41ba-9744-07a1c5a7c20f","Type":"ContainerStarted","Data":"e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579"} Aug 04 13:24:06.887524 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.887484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" event={"ID":"d45c985e-6f19-41ba-9744-07a1c5a7c20f","Type":"ContainerStarted","Data":"db35f26149079c58712ed04f78a447fc83ac5a50b798fd13ca3e7c6621ef496f"} Aug 04 13:24:06.913886 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:06.913839 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podStartSLOduration=1.913823871 podStartE2EDuration="1.913823871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:24:06.911587118 +0000 UTC m=+3377.867182947" watchObservedRunningTime="2026-08-04 13:24:06.913823871 +0000 UTC m=+3377.869419661" Aug 04 13:24:07.248041 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:07.247947 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:07.253330 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:07.253298 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:07.893052 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:07.893024 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:07.894075 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:07.894059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:08.854682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:08.854642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:08.949585 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:08.949554 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:10.018579 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:10.018546 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:10.177507 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:10.177473 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:10.548597 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:10.548563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:10.662490 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:10.662457 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:23.389227 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.388033 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": read tcp 10.134.0.2:34334->10.134.0.79:15021: read: connection reset by peer" start-of-body= Aug 04 13:24:23.389227 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.388102 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": read tcp 10.134.0.2:34334->10.134.0.79:15021: read: connection reset by peer" Aug 04 13:24:23.984387 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.984352 2578 generic.go:356] "Generic (PLEG): container finished" podID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerID="e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579" exitCode=0 Aug 04 13:24:23.984576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.984404 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" event={"ID":"d45c985e-6f19-41ba-9744-07a1c5a7c20f","Type":"ContainerDied","Data":"e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579"} Aug 04 13:24:23.984875 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.984859 2578 scope.go:117] "RemoveContainer" containerID="e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579" Aug 04 13:24:23.985993 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.985964 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:24:23.986063 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.986031 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:24:23.986101 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:23.986062 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:24:24.991354 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:24.991310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" event={"ID":"d45c985e-6f19-41ba-9744-07a1c5a7c20f","Type":"ContainerStarted","Data":"b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e"} Aug 04 13:24:25.247815 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:25.247721 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:25.248105 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:25.248087 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Aug 04 13:24:25.248203 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:25.248153 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Aug 04 13:24:26.097371 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:26.097336 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:26.248748 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:26.248711 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Aug 04 13:24:26.248925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:26.248784 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Aug 04 13:24:27.248517 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:27.248480 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Aug 04 13:24:27.248932 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:27.248562 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Aug 04 13:24:28.248210 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:28.248178 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Aug 04 13:24:28.248406 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:28.248242 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Aug 04 13:24:29.248245 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:29.248215 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Aug 04 13:24:29.248645 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:29.248275 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Aug 04 13:24:30.247950 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:30.247908 2578 patch_prober.go:28] interesting pod/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Aug 04 13:24:30.248167 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:30.247972 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Aug 04 13:24:31.252201 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:31.252165 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:31.252569 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:31.252444 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:31.253304 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:31.253284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:24:33.860132 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:33.860085 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:38.993642 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:38.993603 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:40.730898 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:40.730821 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wjhrj_34420b0a-98d8-4359-b89d-574811fe369d/payload-processing/0.log" Aug 04 13:24:43.939602 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:43.939569 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:24:50.834578 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:24:50.834541 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:30.654500 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:30.654445 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h_664be107-c4e9-4cb5-9d94-522760c54663/payload-processing/0.log" Aug 04 13:25:30.774665 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:30.774633 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wjhrj_34420b0a-98d8-4359-b89d-574811fe369d/payload-processing/0.log" Aug 04 13:25:31.164870 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.164814 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:31.460983 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.460910 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv"] Aug 04 13:25:31.464462 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.464446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.467477 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.467449 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-b88718-openshift-default-dockercfg-n9pc9\"" Aug 04 13:25:31.478292 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.478270 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv"] Aug 04 13:25:31.484575 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.484552 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:31.562365 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.562333 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:31.601227 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-data\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601227 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-socket\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-token\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7dn\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-kube-api-access-rd7dn\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-envoy\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601522 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-credential-socket\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601522 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-certs\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601522 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-podinfo\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.601619 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.601564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1df021b-05b5-4966-b3be-6acd7eec133e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-podinfo\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1df021b-05b5-4966-b3be-6acd7eec133e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-data\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-socket\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-token\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7dn\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-kube-api-access-rd7dn\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-envoy\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-credential-socket\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.702804 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-certs\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.703276 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-data\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.703276 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.702906 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-socket\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.703276 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.703234 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-credential-socket\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.703450 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.703309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1df021b-05b5-4966-b3be-6acd7eec133e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.703450 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.703390 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-certs\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.705090 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.705067 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-podinfo\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.705235 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.705213 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-envoy\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.713876 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.713811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7dn\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-kube-api-access-rd7dn\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.714231 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.714158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-token\") pod \"e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.775533 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.775497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:31.918032 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.918004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv"] Aug 04 13:25:31.920346 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:25:31.920317 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1df021b_05b5_4966_b3be_6acd7eec133e.slice/crio-b949b33fc49c123f9994e4a4eaeffc216deea6e671c6492a1a7a1581b480213f WatchSource:0}: Error finding container b949b33fc49c123f9994e4a4eaeffc216deea6e671c6492a1a7a1581b480213f: Status 404 returned error can't find the container with id b949b33fc49c123f9994e4a4eaeffc216deea6e671c6492a1a7a1581b480213f Aug 04 13:25:31.922711 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.922679 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:25:31.922803 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.922747 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:25:31.922803 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:31.922779 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:25:32.361608 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:32.361564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" event={"ID":"e1df021b-05b5-4966-b3be-6acd7eec133e","Type":"ContainerStarted","Data":"1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0"} Aug 04 13:25:32.361854 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:32.361639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" event={"ID":"e1df021b-05b5-4966-b3be-6acd7eec133e","Type":"ContainerStarted","Data":"b949b33fc49c123f9994e4a4eaeffc216deea6e671c6492a1a7a1581b480213f"} Aug 04 13:25:32.384950 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:32.384877 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podStartSLOduration=1.384858881 podStartE2EDuration="1.384858881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:25:32.382825949 +0000 UTC m=+3463.338421744" watchObservedRunningTime="2026-08-04 13:25:32.384858881 +0000 UTC m=+3463.340454674" Aug 04 13:25:32.775822 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:32.775731 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:32.781279 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:32.781253 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:33.366841 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:33.366806 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:33.367978 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:33.367958 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:34.407668 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:34.407633 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:34.509559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:34.509504 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:35.420314 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:35.420276 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:35.744962 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:35.744877 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:35.946580 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:35.946535 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:36.052200 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.052097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:25:36.053767 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.053743 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7"] Aug 04 13:25:36.057639 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.057624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.060482 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.060457 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b88718\"" Aug 04 13:25:36.060617 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.060517 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b88718-dockercfg-k6rsd\"" Aug 04 13:25:36.069800 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.069759 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7"] Aug 04 13:25:36.150272 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.150240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmww4\" (UniqueName: \"kubernetes.io/projected/58f5124e-5015-4eb9-ad01-eb691f712434-kube-api-access-dmww4\") pod \"payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.150272 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.150275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f5124e-5015-4eb9-ad01-eb691f712434-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.250828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.250791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmww4\" (UniqueName: \"kubernetes.io/projected/58f5124e-5015-4eb9-ad01-eb691f712434-kube-api-access-dmww4\") pod \"payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.250828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.250828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f5124e-5015-4eb9-ad01-eb691f712434-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.251621 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.251603 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f5124e-5015-4eb9-ad01-eb691f712434-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.260701 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.260677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmww4\" (UniqueName: \"kubernetes.io/projected/58f5124e-5015-4eb9-ad01-eb691f712434-kube-api-access-dmww4\") pod \"payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.371247 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.371210 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:36.513578 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:36.513550 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7"] Aug 04 13:25:36.516041 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:25:36.516011 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f5124e_5015_4eb9_ad01_eb691f712434.slice/crio-95aeaf062f0d484ed03357bc9e431f2828e8b3adb9f3ec1e6ceb314870716a96 WatchSource:0}: Error finding container 95aeaf062f0d484ed03357bc9e431f2828e8b3adb9f3ec1e6ceb314870716a96: Status 404 returned error can't find the container with id 95aeaf062f0d484ed03357bc9e431f2828e8b3adb9f3ec1e6ceb314870716a96 Aug 04 13:25:37.389563 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:37.389524 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" event={"ID":"58f5124e-5015-4eb9-ad01-eb691f712434","Type":"ContainerStarted","Data":"5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345"} Aug 04 13:25:37.389713 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:37.389584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" event={"ID":"58f5124e-5015-4eb9-ad01-eb691f712434","Type":"ContainerStarted","Data":"95aeaf062f0d484ed03357bc9e431f2828e8b3adb9f3ec1e6ceb314870716a96"} Aug 04 13:25:37.389713 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:37.389615 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:37.408044 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:37.407999 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" podStartSLOduration=1.407984803 podStartE2EDuration="1.407984803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:25:37.407462696 +0000 UTC m=+3468.363058483" watchObservedRunningTime="2026-08-04 13:25:37.407984803 +0000 UTC m=+3468.363580597" Aug 04 13:25:48.398327 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:48.398278 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" podUID="58f5124e-5015-4eb9-ad01-eb691f712434" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.81:9004: connect: connection refused" Aug 04 13:25:49.408320 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:49.408277 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 13:25:49.408677 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:49.408348 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 13:25:51.481731 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:51.481696 2578 generic.go:356] "Generic (PLEG): container finished" podID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerID="1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0" exitCode=0 Aug 04 13:25:51.482287 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:51.481777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" event={"ID":"e1df021b-05b5-4966-b3be-6acd7eec133e","Type":"ContainerDied","Data":"1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0"} Aug 04 13:25:51.482360 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:51.482328 2578 scope.go:117] "RemoveContainer" containerID="1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0" Aug 04 13:25:51.483508 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:51.483479 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:25:51.483628 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:51.483561 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:25:51.483628 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:51.483613 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 13:25:52.490231 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:52.490192 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" event={"ID":"e1df021b-05b5-4966-b3be-6acd7eec133e","Type":"ContainerStarted","Data":"1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5"} Aug 04 13:25:52.776323 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:52.776233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:52.776532 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:52.776507 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Aug 04 13:25:52.776582 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:52.776559 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Aug 04 13:25:53.776452 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:53.776413 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Aug 04 13:25:53.776845 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:53.776484 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Aug 04 13:25:54.776407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:54.776373 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Aug 04 13:25:54.776589 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:54.776440 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Aug 04 13:25:55.775983 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:55.775949 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Aug 04 13:25:55.776208 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:55.776005 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Aug 04 13:25:56.776114 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:56.776076 2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Aug 04 13:25:56.776516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:56.776148 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Aug 04 13:25:57.780520 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:57.780491 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:57.780969 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:57.780838 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:57.781527 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:57.781512 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:25:58.396668 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:58.396637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:25:59.245778 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:25:59.245735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:05.996186 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:05.996145 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:06.038960 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.038922 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7"] Aug 04 13:26:06.039269 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.039237 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" podUID="58f5124e-5015-4eb9-ad01-eb691f712434" containerName="payload-pre-processing" containerID="cri-o://5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345" gracePeriod=30 Aug 04 13:26:06.066877 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.066843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:06.129931 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.129897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:06.206756 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.206717 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:06.302545 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.302521 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:26:06.379050 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.379015 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f5124e-5015-4eb9-ad01-eb691f712434-plugins-config-volume\") pod \"58f5124e-5015-4eb9-ad01-eb691f712434\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " Aug 04 13:26:06.379276 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.379140 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmww4\" (UniqueName: \"kubernetes.io/projected/58f5124e-5015-4eb9-ad01-eb691f712434-kube-api-access-dmww4\") pod \"58f5124e-5015-4eb9-ad01-eb691f712434\" (UID: \"58f5124e-5015-4eb9-ad01-eb691f712434\") " Aug 04 13:26:06.379457 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.379431 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f5124e-5015-4eb9-ad01-eb691f712434-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58f5124e-5015-4eb9-ad01-eb691f712434" (UID: "58f5124e-5015-4eb9-ad01-eb691f712434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:26:06.379544 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.379526 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f5124e-5015-4eb9-ad01-eb691f712434-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:06.381315 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.381293 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f5124e-5015-4eb9-ad01-eb691f712434-kube-api-access-dmww4" (OuterVolumeSpecName: "kube-api-access-dmww4") pod "58f5124e-5015-4eb9-ad01-eb691f712434" (UID: "58f5124e-5015-4eb9-ad01-eb691f712434"). InnerVolumeSpecName "kube-api-access-dmww4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:26:06.480421 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.480386 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmww4\" (UniqueName: \"kubernetes.io/projected/58f5124e-5015-4eb9-ad01-eb691f712434-kube-api-access-dmww4\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:06.540320 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.540230 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:06.572581 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.572549 2578 generic.go:356] "Generic (PLEG): container finished" podID="58f5124e-5015-4eb9-ad01-eb691f712434" containerID="5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345" exitCode=0 Aug 04 13:26:06.572745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.572603 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" Aug 04 13:26:06.572745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.572641 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" event={"ID":"58f5124e-5015-4eb9-ad01-eb691f712434","Type":"ContainerDied","Data":"5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345"} Aug 04 13:26:06.572745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.572689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7" event={"ID":"58f5124e-5015-4eb9-ad01-eb691f712434","Type":"ContainerDied","Data":"95aeaf062f0d484ed03357bc9e431f2828e8b3adb9f3ec1e6ceb314870716a96"} Aug 04 13:26:06.572745 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.572711 2578 scope.go:117] "RemoveContainer" containerID="5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345" Aug 04 13:26:06.582925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.582906 2578 scope.go:117] "RemoveContainer" containerID="5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345" Aug 04 13:26:06.583245 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:26:06.583226 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345\": container with ID starting with 5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345 not found: ID does not exist" containerID="5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345" Aug 04 13:26:06.583320 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.583252 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345"} err="failed to get container status \"5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345\": rpc error: code = NotFound desc = could not find container \"5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345\": container with ID starting with 5474902ec3d163c835e125522b9c4ea01e3111cee7ea7524d251d9272e1f6345 not found: ID does not exist" Aug 04 13:26:06.597641 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.597616 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7"] Aug 04 13:26:06.601798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:06.601768 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b88718-6f6df5c748-86mk7"] Aug 04 13:26:07.554350 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:07.554317 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f5124e-5015-4eb9-ad01-eb691f712434" path="/var/lib/kubelet/pods/58f5124e-5015-4eb9-ad01-eb691f712434/volumes" Aug 04 13:26:12.989073 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:12.988992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:13.089175 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:13.089114 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:13.242843 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:13.242760 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv"] Aug 04 13:26:13.242998 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:13.242975 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" containerID="cri-o://1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5" gracePeriod=30 Aug 04 13:26:13.296392 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:13.296357 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:13.496689 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:13.496602 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:18.506941 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.506876 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:26:18.603482 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603441 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1df021b-05b5-4966-b3be-6acd7eec133e-istiod-ca-cert\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603501 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-token\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-certs\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603581 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd7dn\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-kube-api-access-rd7dn\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603625 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-credential-socket\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603676 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603668 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-envoy\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603947 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-podinfo\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603947 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603770 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-data\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603947 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603832 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-socket\") pod \"e1df021b-05b5-4966-b3be-6acd7eec133e\" (UID: \"e1df021b-05b5-4966-b3be-6acd7eec133e\") " Aug 04 13:26:18.603947 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603925 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:26:18.604168 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603941 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1df021b-05b5-4966-b3be-6acd7eec133e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:26:18.604168 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.603973 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:26:18.604168 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604153 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:26:18.604335 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604186 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:26:18.604393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604336 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1df021b-05b5-4966-b3be-6acd7eec133e-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.604393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604355 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.604393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604370 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.604393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604383 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.604565 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.604397 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.605922 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.605892 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-kube-api-access-rd7dn" (OuterVolumeSpecName: "kube-api-access-rd7dn") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "kube-api-access-rd7dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:26:18.606060 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.605965 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:26:18.606060 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.606034 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:26:18.606277 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.606243 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1df021b-05b5-4966-b3be-6acd7eec133e" (UID: "e1df021b-05b5-4966-b3be-6acd7eec133e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:26:18.645313 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.645273 2578 generic.go:356] "Generic (PLEG): container finished" podID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerID="1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5" exitCode=0 Aug 04 13:26:18.645540 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.645332 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" Aug 04 13:26:18.645540 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.645326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" event={"ID":"e1df021b-05b5-4966-b3be-6acd7eec133e","Type":"ContainerDied","Data":"1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5"} Aug 04 13:26:18.645540 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.645402 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv" event={"ID":"e1df021b-05b5-4966-b3be-6acd7eec133e","Type":"ContainerDied","Data":"b949b33fc49c123f9994e4a4eaeffc216deea6e671c6492a1a7a1581b480213f"} Aug 04 13:26:18.645540 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.645432 2578 scope.go:117] "RemoveContainer" containerID="1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5" Aug 04 13:26:18.657460 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.657434 2578 scope.go:117] "RemoveContainer" containerID="1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0" Aug 04 13:26:18.667071 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.667046 2578 scope.go:117] "RemoveContainer" containerID="1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5" Aug 04 13:26:18.667417 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:26:18.667396 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5\": container with ID starting with 1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5 not found: ID does not exist" containerID="1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5" Aug 04 13:26:18.667474 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.667428 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5"} err="failed to get container status \"1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5\": rpc error: code = NotFound desc = could not find container \"1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5\": container with ID starting with 1058a336386cce1450ec11a2142b57ac896ead549deab8d1b50f54575f1f29b5 not found: ID does not exist" Aug 04 13:26:18.667474 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.667448 2578 scope.go:117] "RemoveContainer" containerID="1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0" Aug 04 13:26:18.667714 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:26:18.667695 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0\": container with ID starting with 1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0 not found: ID does not exist" containerID="1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0" Aug 04 13:26:18.667765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.667720 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0"} err="failed to get container status \"1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0\": rpc error: code = NotFound desc = could not find container \"1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0\": container with ID starting with 1a1bf5838f04b2302957125bc0dd6520a4276bcc18e9cdc0374cb3faced220e0 not found: ID does not exist" Aug 04 13:26:18.672813 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.672789 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv"] Aug 04 13:26:18.677223 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.677198 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-b88718-openshift-default-5d47c558bd-jjtlv"] Aug 04 13:26:18.705492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.705459 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.705492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.705485 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd7dn\" (UniqueName: \"kubernetes.io/projected/e1df021b-05b5-4966-b3be-6acd7eec133e-kube-api-access-rd7dn\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.705492 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.705496 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:18.705706 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:18.705505 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1df021b-05b5-4966-b3be-6acd7eec133e-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:19.553642 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:19.553606 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" path="/var/lib/kubelet/pods/e1df021b-05b5-4966-b3be-6acd7eec133e/volumes" Aug 04 13:26:32.730332 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.730286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:32.770916 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.770879 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h"] Aug 04 13:26:32.771587 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.771531 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" podUID="664be107-c4e9-4cb5-9d94-522760c54663" containerName="payload-processing" containerID="cri-o://3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee" gracePeriod=30 Aug 04 13:26:32.780591 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.780556 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb"] Aug 04 13:26:32.781065 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.781019 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" podUID="4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" containerName="payload-pre-processing" containerID="cri-o://bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99" gracePeriod=30 Aug 04 13:26:32.822407 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.822359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:32.882663 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.882627 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:32.949031 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:32.948914 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:33.052002 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.051966 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:26:33.085735 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.085709 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:26:33.149594 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.149560 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p74tp\" (UniqueName: \"kubernetes.io/projected/664be107-c4e9-4cb5-9d94-522760c54663-kube-api-access-p74tp\") pod \"664be107-c4e9-4cb5-9d94-522760c54663\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " Aug 04 13:26:33.149765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.149695 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664be107-c4e9-4cb5-9d94-522760c54663-plugins-config-volume\") pod \"664be107-c4e9-4cb5-9d94-522760c54663\" (UID: \"664be107-c4e9-4cb5-9d94-522760c54663\") " Aug 04 13:26:33.150028 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.149995 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664be107-c4e9-4cb5-9d94-522760c54663-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "664be107-c4e9-4cb5-9d94-522760c54663" (UID: "664be107-c4e9-4cb5-9d94-522760c54663"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:26:33.150199 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.150066 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664be107-c4e9-4cb5-9d94-522760c54663-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:33.151940 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.151912 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664be107-c4e9-4cb5-9d94-522760c54663-kube-api-access-p74tp" (OuterVolumeSpecName: "kube-api-access-p74tp") pod "664be107-c4e9-4cb5-9d94-522760c54663" (UID: "664be107-c4e9-4cb5-9d94-522760c54663"). InnerVolumeSpecName "kube-api-access-p74tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:26:33.251353 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.251307 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8kcs\" (UniqueName: \"kubernetes.io/projected/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-kube-api-access-p8kcs\") pod \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " Aug 04 13:26:33.251556 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.251370 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-plugins-config-volume\") pod \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\" (UID: \"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5\") " Aug 04 13:26:33.251764 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.251740 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" (UID: "4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:26:33.251828 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.251762 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74tp\" (UniqueName: \"kubernetes.io/projected/664be107-c4e9-4cb5-9d94-522760c54663-kube-api-access-p74tp\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:33.253646 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.253625 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-kube-api-access-p8kcs" (OuterVolumeSpecName: "kube-api-access-p8kcs") pod "4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" (UID: "4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5"). InnerVolumeSpecName "kube-api-access-p8kcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:26:33.329830 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.329746 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:26:33.352724 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.352684 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kcs\" (UniqueName: \"kubernetes.io/projected/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-kube-api-access-p8kcs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:33.352724 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.352717 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:26:33.730262 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.730224 2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" containerID="bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99" exitCode=0 Aug 04 13:26:33.730466 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.730279 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" Aug 04 13:26:33.730466 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.730309 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" event={"ID":"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5","Type":"ContainerDied","Data":"bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99"} Aug 04 13:26:33.730466 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.730396 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb" event={"ID":"4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5","Type":"ContainerDied","Data":"34d2c28864625691865c76962acdf0fcb70d6a0da6485749a0d130f32aaa9394"} Aug 04 13:26:33.730466 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.730426 2578 scope.go:117] "RemoveContainer" containerID="bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99" Aug 04 13:26:33.731700 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.731669 2578 generic.go:356] "Generic (PLEG): container finished" podID="664be107-c4e9-4cb5-9d94-522760c54663" containerID="3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee" exitCode=0 Aug 04 13:26:33.731798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.731735 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" Aug 04 13:26:33.731798 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.731754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" event={"ID":"664be107-c4e9-4cb5-9d94-522760c54663","Type":"ContainerDied","Data":"3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee"} Aug 04 13:26:33.731884 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.731815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h" event={"ID":"664be107-c4e9-4cb5-9d94-522760c54663","Type":"ContainerDied","Data":"0526c72f6c7305009bf26a98c13c115e20dcff915281e8220a10be331b99f3d5"} Aug 04 13:26:33.739975 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.739954 2578 scope.go:117] "RemoveContainer" containerID="bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99" Aug 04 13:26:33.740278 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:26:33.740260 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99\": container with ID starting with bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99 not found: ID does not exist" containerID="bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99" Aug 04 13:26:33.740329 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.740295 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99"} err="failed to get container status \"bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99\": rpc error: code = NotFound desc = could not find container \"bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99\": container with ID starting with bcda717aec537f33ce2a3d0c8b69bbfb8bd60eed608a6302eeefcdffc2d82f99 not found: ID does not exist" Aug 04 13:26:33.740329 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.740312 2578 scope.go:117] "RemoveContainer" containerID="3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee" Aug 04 13:26:33.749314 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.749286 2578 scope.go:117] "RemoveContainer" containerID="3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee" Aug 04 13:26:33.749635 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:26:33.749612 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee\": container with ID starting with 3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee not found: ID does not exist" containerID="3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee" Aug 04 13:26:33.749687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.749645 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee"} err="failed to get container status \"3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee\": rpc error: code = NotFound desc = could not find container \"3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee\": container with ID starting with 3fde954e5f072162f5b82e2095bdf979e091b6bf4e3ed2623968c43c4401b9ee not found: ID does not exist" Aug 04 13:26:33.750603 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.750583 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h"] Aug 04 13:26:33.754705 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.754677 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ef922-59978bd47d-vjq9h"] Aug 04 13:26:33.766597 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.766568 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb"] Aug 04 13:26:33.772193 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:33.772162 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2ef922-7947dd95c6-5wlcb"] Aug 04 13:26:35.554938 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:35.554904 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" path="/var/lib/kubelet/pods/4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5/volumes" Aug 04 13:26:35.555348 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:35.555293 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="664be107-c4e9-4cb5-9d94-522760c54663" path="/var/lib/kubelet/pods/664be107-c4e9-4cb5-9d94-522760c54663/volumes" Aug 04 13:26:38.409637 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:26:38.409598 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:07.613957 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:07.613919 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:14.513078 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:14.513034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:16.745858 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:16.745799 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:17.838457 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:17.838427 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:17.932881 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:17.932831 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:23.148371 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:23.148332 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:23.214152 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:23.214067 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:23.439920 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:23.439816 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:23.694925 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:23.694810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:29.296448 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:29.296403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:29.399769 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:29.399737 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:29.497576 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:29.497545 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:29.566378 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:29.566292 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:29.949248 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:29.949211 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:37.143423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:37.143390 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:37.208607 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:37.208569 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:37.384488 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:37.384448 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn"] Aug 04 13:27:37.384735 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:37.384685 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" containerID="cri-o://b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e" gracePeriod=30 Aug 04 13:27:37.426187 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:37.426054 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:37.670858 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:37.670802 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:42.631930 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.631902 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:27:42.740341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740257 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-socket\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740341 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740301 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istiod-ca-cert\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740368 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxj24\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-kube-api-access-qxj24\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-certs\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740439 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-podinfo\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-credential-socket\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740534 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740504 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-envoy\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740538 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-data\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740581 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:27:42.740765 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740617 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-token\") pod \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\" (UID: \"d45c985e-6f19-41ba-9744-07a1c5a7c20f\") " Aug 04 13:27:42.740922 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740805 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:27:42.740995 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740954 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:27:42.741048 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.740988 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:27:42.741167 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.741103 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.741167 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.741169 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.741542 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.741184 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.741542 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.741197 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.741542 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.741380 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:27:42.742937 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.742911 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:27:42.743035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.742928 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:27:42.743035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.742948 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-kube-api-access-qxj24" (OuterVolumeSpecName: "kube-api-access-qxj24") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "kube-api-access-qxj24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:27:42.743035 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.743008 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d45c985e-6f19-41ba-9744-07a1c5a7c20f" (UID: "d45c985e-6f19-41ba-9744-07a1c5a7c20f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:27:42.842483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.842442 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxj24\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-kube-api-access-qxj24\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.842483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.842476 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.842483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.842485 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.842483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.842493 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:42.842759 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:42.842500 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45c985e-6f19-41ba-9744-07a1c5a7c20f-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:43.102727 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.102692 2578 generic.go:356] "Generic (PLEG): container finished" podID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerID="b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e" exitCode=0 Aug 04 13:27:43.102896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.102752 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" Aug 04 13:27:43.102896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.102788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" event={"ID":"d45c985e-6f19-41ba-9744-07a1c5a7c20f","Type":"ContainerDied","Data":"b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e"} Aug 04 13:27:43.102896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.102847 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn" event={"ID":"d45c985e-6f19-41ba-9744-07a1c5a7c20f","Type":"ContainerDied","Data":"db35f26149079c58712ed04f78a447fc83ac5a50b798fd13ca3e7c6621ef496f"} Aug 04 13:27:43.102896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.102870 2578 scope.go:117] "RemoveContainer" containerID="b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e" Aug 04 13:27:43.112868 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.112840 2578 scope.go:117] "RemoveContainer" containerID="e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579" Aug 04 13:27:43.122168 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.122141 2578 scope.go:117] "RemoveContainer" containerID="b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e" Aug 04 13:27:43.122478 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:27:43.122457 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e\": container with ID starting with b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e not found: ID does not exist" containerID="b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e" Aug 04 13:27:43.122577 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.122493 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e"} err="failed to get container status \"b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e\": rpc error: code = NotFound desc = could not find container \"b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e\": container with ID starting with b9615e9394cc1493cf680d5a24d711afce5ae593baaf46795451d051a3baa25e not found: ID does not exist" Aug 04 13:27:43.122577 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.122518 2578 scope.go:117] "RemoveContainer" containerID="e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579" Aug 04 13:27:43.122783 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:27:43.122765 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579\": container with ID starting with e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579 not found: ID does not exist" containerID="e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579" Aug 04 13:27:43.122823 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.122789 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579"} err="failed to get container status \"e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579\": rpc error: code = NotFound desc = could not find container \"e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579\": container with ID starting with e2e92207a8b18719a36f8bd5f9fe42ac59a8e090afac52ed9d016f987c091579 not found: ID does not exist" Aug 04 13:27:43.127495 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.127455 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn"] Aug 04 13:27:43.131856 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.131829 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-77ce5c-openshift-default-5d67b5f776-k5sfn"] Aug 04 13:27:43.553282 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:43.553203 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" path="/var/lib/kubelet/pods/d45c985e-6f19-41ba-9744-07a1c5a7c20f/volumes" Aug 04 13:27:48.466906 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.466867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:48.507357 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.507322 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft"] Aug 04 13:27:48.507670 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.507622 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" podUID="80fedacc-35c3-4873-a2c8-151e6804b11b" containerName="payload-pre-processing" containerID="cri-o://083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e" gracePeriod=30 Aug 04 13:27:48.562767 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.562732 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:48.720157 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.720055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:48.772682 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.772656 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:27:48.810945 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.810848 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:48.904390 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.904352 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w75pw\" (UniqueName: \"kubernetes.io/projected/80fedacc-35c3-4873-a2c8-151e6804b11b-kube-api-access-w75pw\") pod \"80fedacc-35c3-4873-a2c8-151e6804b11b\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " Aug 04 13:27:48.904571 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.904419 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fedacc-35c3-4873-a2c8-151e6804b11b-plugins-config-volume\") pod \"80fedacc-35c3-4873-a2c8-151e6804b11b\" (UID: \"80fedacc-35c3-4873-a2c8-151e6804b11b\") " Aug 04 13:27:48.904845 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.904813 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fedacc-35c3-4873-a2c8-151e6804b11b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80fedacc-35c3-4873-a2c8-151e6804b11b" (UID: "80fedacc-35c3-4873-a2c8-151e6804b11b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:27:48.906660 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:48.906637 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fedacc-35c3-4873-a2c8-151e6804b11b-kube-api-access-w75pw" (OuterVolumeSpecName: "kube-api-access-w75pw") pod "80fedacc-35c3-4873-a2c8-151e6804b11b" (UID: "80fedacc-35c3-4873-a2c8-151e6804b11b"). InnerVolumeSpecName "kube-api-access-w75pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:27:49.005488 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.005395 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w75pw\" (UniqueName: \"kubernetes.io/projected/80fedacc-35c3-4873-a2c8-151e6804b11b-kube-api-access-w75pw\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:49.005488 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.005431 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fedacc-35c3-4873-a2c8-151e6804b11b-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:27:49.135328 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.135292 2578 generic.go:356] "Generic (PLEG): container finished" podID="80fedacc-35c3-4873-a2c8-151e6804b11b" containerID="083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e" exitCode=0 Aug 04 13:27:49.135505 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.135338 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" Aug 04 13:27:49.135505 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.135381 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" event={"ID":"80fedacc-35c3-4873-a2c8-151e6804b11b","Type":"ContainerDied","Data":"083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e"} Aug 04 13:27:49.135505 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.135435 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft" event={"ID":"80fedacc-35c3-4873-a2c8-151e6804b11b","Type":"ContainerDied","Data":"3ffdc65ef94251f8e6232a65cd99bb84a5e3efd87284100132effc99962f552d"} Aug 04 13:27:49.135505 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.135463 2578 scope.go:117] "RemoveContainer" containerID="083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e" Aug 04 13:27:49.146917 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.146308 2578 scope.go:117] "RemoveContainer" containerID="083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e" Aug 04 13:27:49.147330 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:27:49.147305 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e\": container with ID starting with 083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e not found: ID does not exist" containerID="083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e" Aug 04 13:27:49.147422 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.147337 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e"} err="failed to get container status \"083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e\": rpc error: code = NotFound desc = could not find container \"083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e\": container with ID starting with 083fe1e7087125009f51ebf44dabb2f14f4ee15322158c9ed558f49e7195008e not found: ID does not exist" Aug 04 13:27:49.168687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.168659 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft"] Aug 04 13:27:49.170403 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.170383 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6a00c1-54cbbc5448-blfft"] Aug 04 13:27:49.173036 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.173014 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:49.552998 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.552962 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80fedacc-35c3-4873-a2c8-151e6804b11b" path="/var/lib/kubelet/pods/80fedacc-35c3-4873-a2c8-151e6804b11b/volumes" Aug 04 13:27:49.767689 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.767663 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:27:49.775052 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:49.775027 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qch7k_72ca687e-554b-41b0-9feb-40167b647220/ovn-acl-logging/0.log" Aug 04 13:27:55.712862 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:55.712825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:55.814231 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:55.814201 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:55.973605 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:55.973514 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs"] Aug 04 13:27:55.973794 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:55.973768 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" containerID="cri-o://288fbb3d33d67f68613cb923ad09d99e642752d9b77499efe62e61ab9b133a45" gracePeriod=30 Aug 04 13:27:55.993401 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:55.993370 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:27:56.201019 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:27:56.200982 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:00.018888 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:00.018843 2578 patch_prober.go:28] interesting pod/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 13:28:00.019296 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:00.018927 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 13:28:01.031467 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:28:01.031432 2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2b2e22_81bf_4189_9af0_0abed7af5230.slice/crio-conmon-288fbb3d33d67f68613cb923ad09d99e642752d9b77499efe62e61ab9b133a45.scope\": RecentStats: unable to find data in memory cache]" Aug 04 13:28:01.197747 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.197716 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerID="288fbb3d33d67f68613cb923ad09d99e642752d9b77499efe62e61ab9b133a45" exitCode=0 Aug 04 13:28:01.197896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.197795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" event={"ID":"cf2b2e22-81bf-4189-9af0-0abed7af5230","Type":"ContainerDied","Data":"288fbb3d33d67f68613cb923ad09d99e642752d9b77499efe62e61ab9b133a45"} Aug 04 13:28:01.197896 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.197888 2578 scope.go:117] "RemoveContainer" containerID="1b08fa688264bfbb7883e025953a5c301741c5dbc3cbfddae0e40e43f43ebcd4" Aug 04 13:28:01.229833 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.229812 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:28:01.331699 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-envoy\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.331699 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-credential-socket\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.331699 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331679 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-certs\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.331964 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331716 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-data\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.331964 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331747 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77dlq\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-kube-api-access-77dlq\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.331964 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331858 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-podinfo\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.331964 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331891 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-token\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.332113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.331959 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:01.332113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332008 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b2e22-81bf-4189-9af0-0abed7af5230-istiod-ca-cert\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.332113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332052 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:01.332113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332066 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-socket\") pod \"cf2b2e22-81bf-4189-9af0-0abed7af5230\" (UID: \"cf2b2e22-81bf-4189-9af0-0abed7af5230\") " Aug 04 13:28:01.332113 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332070 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:01.332455 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332352 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:01.332529 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332467 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2b2e22-81bf-4189-9af0-0abed7af5230-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:28:01.332529 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332520 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.332626 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332538 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.332626 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332550 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.332626 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.332560 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.334096 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.334075 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:01.334359 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.334328 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-kube-api-access-77dlq" (OuterVolumeSpecName: "kube-api-access-77dlq") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "kube-api-access-77dlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:28:01.334655 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.334627 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:28:01.334732 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.334683 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf2b2e22-81bf-4189-9af0-0abed7af5230" (UID: "cf2b2e22-81bf-4189-9af0-0abed7af5230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:28:01.433234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.433196 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.433234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.433227 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77dlq\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-kube-api-access-77dlq\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.433234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.433237 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.433234 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.433245 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf2b2e22-81bf-4189-9af0-0abed7af5230-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:01.433504 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:01.433253 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf2b2e22-81bf-4189-9af0-0abed7af5230-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:02.204404 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:02.204369 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" Aug 04 13:28:02.204874 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:02.204407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs" event={"ID":"cf2b2e22-81bf-4189-9af0-0abed7af5230","Type":"ContainerDied","Data":"cd2bd103748ce01a268a65243218a35cda41d071acd9b3360dbaf6991f4d784a"} Aug 04 13:28:02.204874 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:02.204477 2578 scope.go:117] "RemoveContainer" containerID="288fbb3d33d67f68613cb923ad09d99e642752d9b77499efe62e61ab9b133a45" Aug 04 13:28:02.227954 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:02.227919 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs"] Aug 04 13:28:02.233537 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:02.233509 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6a00c1-openshift-default-7bbdd9f4d8-c7sxs"] Aug 04 13:28:03.553332 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:03.553298 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" path="/var/lib/kubelet/pods/cf2b2e22-81bf-4189-9af0-0abed7af5230/volumes" Aug 04 13:28:06.876838 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:06.876803 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:06.919524 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:06.919492 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g"] Aug 04 13:28:06.919775 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:06.919728 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" podUID="d04e3ba8-7868-405c-b444-5e8951fee570" containerName="payload-pre-processing" containerID="cri-o://d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f" gracePeriod=30 Aug 04 13:28:06.946156 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:06.946113 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:07.012228 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.012176 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:07.082163 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.082099 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:07.173471 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.173444 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:28:07.231921 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.231887 2578 generic.go:356] "Generic (PLEG): container finished" podID="d04e3ba8-7868-405c-b444-5e8951fee570" containerID="d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f" exitCode=0 Aug 04 13:28:07.232108 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.231931 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" Aug 04 13:28:07.232108 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.231946 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" event={"ID":"d04e3ba8-7868-405c-b444-5e8951fee570","Type":"ContainerDied","Data":"d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f"} Aug 04 13:28:07.232108 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.232005 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g" event={"ID":"d04e3ba8-7868-405c-b444-5e8951fee570","Type":"ContainerDied","Data":"6a338ba671956cebd8271a2b22cefb1ae459461792a4aed7124bcf8c630b0118"} Aug 04 13:28:07.232108 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.232038 2578 scope.go:117] "RemoveContainer" containerID="d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f" Aug 04 13:28:07.240733 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.240715 2578 scope.go:117] "RemoveContainer" containerID="d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f" Aug 04 13:28:07.241008 ip-10-0-136-46 kubenswrapper[2578]: E0804 13:28:07.240987 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f\": container with ID starting with d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f not found: ID does not exist" containerID="d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f" Aug 04 13:28:07.241065 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.241020 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f"} err="failed to get container status \"d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f\": rpc error: code = NotFound desc = could not find container \"d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f\": container with ID starting with d5adc6765d819ce37d431708c3781fc150ffa3b86f86d6b1cead5bd922dd738f not found: ID does not exist" Aug 04 13:28:07.290495 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.290465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btv9n\" (UniqueName: \"kubernetes.io/projected/d04e3ba8-7868-405c-b444-5e8951fee570-kube-api-access-btv9n\") pod \"d04e3ba8-7868-405c-b444-5e8951fee570\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " Aug 04 13:28:07.290637 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.290528 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04e3ba8-7868-405c-b444-5e8951fee570-plugins-config-volume\") pod \"d04e3ba8-7868-405c-b444-5e8951fee570\" (UID: \"d04e3ba8-7868-405c-b444-5e8951fee570\") " Aug 04 13:28:07.290934 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.290911 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04e3ba8-7868-405c-b444-5e8951fee570-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d04e3ba8-7868-405c-b444-5e8951fee570" (UID: "d04e3ba8-7868-405c-b444-5e8951fee570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:28:07.292713 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.292694 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04e3ba8-7868-405c-b444-5e8951fee570-kube-api-access-btv9n" (OuterVolumeSpecName: "kube-api-access-btv9n") pod "d04e3ba8-7868-405c-b444-5e8951fee570" (UID: "d04e3ba8-7868-405c-b444-5e8951fee570"). InnerVolumeSpecName "kube-api-access-btv9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:28:07.391474 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.391439 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv9n\" (UniqueName: \"kubernetes.io/projected/d04e3ba8-7868-405c-b444-5e8951fee570-kube-api-access-btv9n\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:07.391474 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.391469 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04e3ba8-7868-405c-b444-5e8951fee570-plugins-config-volume\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:07.473516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.473440 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:07.556055 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.556026 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g"] Aug 04 13:28:07.560139 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:07.560099 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c1c995-5d56bfddb4-2fk9g"] Aug 04 13:28:09.552908 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:09.552868 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d04e3ba8-7868-405c-b444-5e8951fee570" path="/var/lib/kubelet/pods/d04e3ba8-7868-405c-b444-5e8951fee570/volumes" Aug 04 13:28:13.842111 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:13.842075 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:13.904884 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:13.904836 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:14.078238 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:14.078203 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs"] Aug 04 13:28:14.078519 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:14.078493 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" containerID="cri-o://9c2442bcbbe4fd1f26f17285e1146294a9f9fd0882d44e2e86bfb728bc66223a" gracePeriod=30 Aug 04 13:28:14.101148 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:14.101036 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:14.322737 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:14.322690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:14.740984 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:14.740951 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55d4964dc4-nq6lb_243410bc-221c-4bb5-8a52-6624a11c1f97/authorino/0.log" Aug 04 13:28:19.297973 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.297938 2578 generic.go:356] "Generic (PLEG): container finished" podID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerID="9c2442bcbbe4fd1f26f17285e1146294a9f9fd0882d44e2e86bfb728bc66223a" exitCode=0 Aug 04 13:28:19.298504 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.298011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" event={"ID":"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa","Type":"ContainerDied","Data":"9c2442bcbbe4fd1f26f17285e1146294a9f9fd0882d44e2e86bfb728bc66223a"} Aug 04 13:28:19.298504 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.298076 2578 scope.go:117] "RemoveContainer" containerID="70743cf614dcedfe6f9a7e7a1d770d54a8c035879b28209e23186a86376fc118" Aug 04 13:28:19.333389 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.333363 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:28:19.409277 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409197 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istiod-ca-cert\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409277 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409241 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9vfj\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-kube-api-access-d9vfj\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409277 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409264 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-credential-socket\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409535 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409294 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-envoy\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409535 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409475 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-podinfo\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409535 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409526 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-socket\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409692 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409574 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-data\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409692 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409621 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-certs\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409692 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409641 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:19.409692 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409671 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 13:28:19.409692 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409681 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-token\") pod \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\" (UID: \"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa\") " Aug 04 13:28:19.409930 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409861 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:19.409930 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409858 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:19.409930 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.409875 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:19.410345 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.410324 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istiod-ca-cert\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.410444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.410351 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-credential-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.410444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.410366 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-socket\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.410444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.410379 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-data\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.410444 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.410391 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-workload-certs\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.411623 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.411597 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 13:28:19.412065 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.412033 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:28:19.412193 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.412135 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-kube-api-access-d9vfj" (OuterVolumeSpecName: "kube-api-access-d9vfj") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "kube-api-access-d9vfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 13:28:19.412193 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.412143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" (UID: "1b00cdcc-d8d5-43cc-86e0-1c243eb411fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 13:28:19.510944 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.510909 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9vfj\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-kube-api-access-d9vfj\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.510944 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.510939 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-envoy\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.510944 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.510949 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-podinfo\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:19.511210 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:19.510957 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa-istio-token\") on node \"ip-10-0-136-46.ec2.internal\" DevicePath \"\"" Aug 04 13:28:20.144753 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.144720 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68d8768cdd-c7g2g_b6598625-e743-4b38-8ef5-7f62371e40e8/manager/0.log" Aug 04 13:28:20.305279 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.305247 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" Aug 04 13:28:20.305687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.305239 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs" event={"ID":"1b00cdcc-d8d5-43cc-86e0-1c243eb411fa","Type":"ContainerDied","Data":"33e29d32f87febdcbf1186dfdc8fa5d1dd2c1f48be3e6d3d3b6be13eef0902c8"} Aug 04 13:28:20.305687 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.305379 2578 scope.go:117] "RemoveContainer" containerID="9c2442bcbbe4fd1f26f17285e1146294a9f9fd0882d44e2e86bfb728bc66223a" Aug 04 13:28:20.327266 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.327226 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs"] Aug 04 13:28:20.329938 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.329911 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c1c995-openshift-default-5598cb4764-8wvxs"] Aug 04 13:28:20.734693 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.734659 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547f69df5b-9lrrc_3654256c-6789-438c-97f0-08d6c2ef0be2/manager/0.log" Aug 04 13:28:20.741656 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:20.741638 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547f69df5b-9lrrc_3654256c-6789-438c-97f0-08d6c2ef0be2/copy-manifests/0.log" Aug 04 13:28:21.076253 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:21.076178 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8544796ddc-h6bqb_4fd5b390-56a1-4127-b4fd-7631f66c54bc/maas-api/0.log" Aug 04 13:28:21.194198 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:21.194157 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764125-grz8t_442abb3b-6308-473c-8864-2f86d1e65670/cleanup/0.log" Aug 04 13:28:21.553288 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:21.553253 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" path="/var/lib/kubelet/pods/1b00cdcc-d8d5-43cc-86e0-1c243eb411fa/volumes" Aug 04 13:28:22.285258 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:22.285220 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55d4964dc4-nq6lb_243410bc-221c-4bb5-8a52-6624a11c1f97/authorino/0.log" Aug 04 13:28:22.600223 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:22.600198 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vcpks_c011127a-ba86-472f-bf21-4585f9e9e1d4/kuadrant-console-plugin/0.log" Aug 04 13:28:23.664987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:23.664955 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688595d9d-hvkqz_a3bce0e2-a1cc-4759-a048-cf707c9ffe66/kube-auth-proxy/0.log" Aug 04 13:28:24.004068 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:24.003977 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wjhrj_34420b0a-98d8-4359-b89d-574811fe369d/payload-processing/0.log" Aug 04 13:28:24.459996 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:24.459966 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tncv6_388c7314-2222-425e-952b-47ca549d65fe/main/0.log" Aug 04 13:28:24.566423 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:24.566392 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-f7bk7_d8c4fb28-fc80-4bd7-90a4-a46ff68e7c0e/main/0.log" Aug 04 13:28:30.460417 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460382 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kqx2z/must-gather-bshqq"] Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460765 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460777 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460792 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460797 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460805 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460811 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460818 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" containerName="payload-pre-processing" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460824 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" containerName="payload-pre-processing" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460832 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" Aug 04 13:28:30.460835 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460838 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460844 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460849 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460859 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460866 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460871 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f5124e-5015-4eb9-ad01-eb691f712434" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460877 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f5124e-5015-4eb9-ad01-eb691f712434" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460885 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fedacc-35c3-4873-a2c8-151e6804b11b" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460890 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fedacc-35c3-4873-a2c8-151e6804b11b" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460896 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460902 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460911 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04e3ba8-7868-405c-b444-5e8951fee570" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460915 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04e3ba8-7868-405c-b444-5e8951fee570" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460921 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="664be107-c4e9-4cb5-9d94-522760c54663" containerName="payload-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460926 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="664be107-c4e9-4cb5-9d94-522760c54663" containerName="payload-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460983 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="664be107-c4e9-4cb5-9d94-522760c54663" containerName="payload-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460990 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea01e1c-2ded-440e-bc5a-7f22ccc1e7a5" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.460996 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461002 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461009 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461014 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b00cdcc-d8d5-43cc-86e0-1c243eb411fa" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461020 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04e3ba8-7868-405c-b444-5e8951fee570" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461026 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f5124e-5015-4eb9-ad01-eb691f712434" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461032 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fedacc-35c3-4873-a2c8-151e6804b11b" containerName="payload-pre-processing" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461038 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461043 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1df021b-05b5-4966-b3be-6acd7eec133e" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461102 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461107 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461183 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf2b2e22-81bf-4189-9af0-0abed7af5230" containerName="istio-proxy" Aug 04 13:28:30.461278 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.461190 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d45c985e-6f19-41ba-9744-07a1c5a7c20f" containerName="istio-proxy" Aug 04 13:28:30.464968 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.464941 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:30.465755 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.465740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.468898 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.468877 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqx2z\"/\"kube-root-ca.crt\"" Aug 04 13:28:30.469275 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.469259 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kqx2z\"/\"openshift-service-ca.crt\"" Aug 04 13:28:30.476162 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.476140 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqx2z/must-gather-bshqq"] Aug 04 13:28:30.534246 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.534207 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" limit=1024 Aug 04 13:28:30.614091 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.614054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/585042b6-0cab-4552-addc-3cfd60857198-must-gather-output\") pod \"must-gather-bshqq\" (UID: \"585042b6-0cab-4552-addc-3cfd60857198\") " pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.614296 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.614230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlkz\" (UniqueName: \"kubernetes.io/projected/585042b6-0cab-4552-addc-3cfd60857198-kube-api-access-wmlkz\") pod \"must-gather-bshqq\" (UID: \"585042b6-0cab-4552-addc-3cfd60857198\") " pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.715494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.715408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlkz\" (UniqueName: \"kubernetes.io/projected/585042b6-0cab-4552-addc-3cfd60857198-kube-api-access-wmlkz\") pod \"must-gather-bshqq\" (UID: \"585042b6-0cab-4552-addc-3cfd60857198\") " pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.715494 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.715453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/585042b6-0cab-4552-addc-3cfd60857198-must-gather-output\") pod \"must-gather-bshqq\" (UID: \"585042b6-0cab-4552-addc-3cfd60857198\") " pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.715858 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.715839 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/585042b6-0cab-4552-addc-3cfd60857198-must-gather-output\") pod \"must-gather-bshqq\" (UID: \"585042b6-0cab-4552-addc-3cfd60857198\") " pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.725283 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.725264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlkz\" (UniqueName: \"kubernetes.io/projected/585042b6-0cab-4552-addc-3cfd60857198-kube-api-access-wmlkz\") pod \"must-gather-bshqq\" (UID: \"585042b6-0cab-4552-addc-3cfd60857198\") " pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.776516 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.776488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqx2z/must-gather-bshqq" Aug 04 13:28:30.913040 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.913017 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqx2z/must-gather-bshqq"] Aug 04 13:28:30.915336 ip-10-0-136-46 kubenswrapper[2578]: W0804 13:28:30.915294 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod585042b6_0cab_4552_addc_3cfd60857198.slice/crio-740c4f735d6bbe952eab538d452711ababbd2f078f571cf1c44e5a3831fa845c WatchSource:0}: Error finding container 740c4f735d6bbe952eab538d452711ababbd2f078f571cf1c44e5a3831fa845c: Status 404 returned error can't find the container with id 740c4f735d6bbe952eab538d452711ababbd2f078f571cf1c44e5a3831fa845c Aug 04 13:28:30.917031 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:30.917011 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 13:28:31.363539 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:31.363503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqx2z/must-gather-bshqq" event={"ID":"585042b6-0cab-4552-addc-3cfd60857198","Type":"ContainerStarted","Data":"740c4f735d6bbe952eab538d452711ababbd2f078f571cf1c44e5a3831fa845c"} Aug 04 13:28:32.372339 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:32.372302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqx2z/must-gather-bshqq" event={"ID":"585042b6-0cab-4552-addc-3cfd60857198","Type":"ContainerStarted","Data":"b12e78d5c3facc37cddfbfcd08a618a682826a27e86f9db1d0b807e86009bfd0"} Aug 04 13:28:32.372815 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:32.372355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqx2z/must-gather-bshqq" event={"ID":"585042b6-0cab-4552-addc-3cfd60857198","Type":"ContainerStarted","Data":"c7cc3cdb2663bb0f97236f30969b03bdd8d977e4751e23fd1ed3b38b20b251ac"} Aug 04 13:28:32.392343 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:32.392275 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kqx2z/must-gather-bshqq" podStartSLOduration=1.4779512000000001 podStartE2EDuration="2.392254511s" totalImagesPullingTime="914.303311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 13:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:28:32.390185549 +0000 UTC m=+3643.345781335" watchObservedRunningTime="2026-08-04 13:28:32.392254511 +0000 UTC m=+3643.347850306" Aug 04 13:28:33.552483 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:33.552454 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kd7rr_c2f3b6b1-86db-4fb6-9afd-e305159d16a2/global-pull-secret-syncer/0.log" Aug 04 13:28:33.655945 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:33.655915 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8j5qs_7d82696e-2556-495c-93d3-4b882156c2a4/konnectivity-agent/0.log" Aug 04 13:28:33.832694 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:33.832620 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-46.ec2.internal_1a2944b9797870016f92974beffaabc3/haproxy/0.log" Aug 04 13:28:38.601022 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:38.598704 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55d4964dc4-nq6lb_243410bc-221c-4bb5-8a52-6624a11c1f97/authorino/0.log" Aug 04 13:28:38.713038 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:38.713000 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vcpks_c011127a-ba86-472f-bf21-4585f9e9e1d4/kuadrant-console-plugin/0.log" Aug 04 13:28:40.466559 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.466508 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/alertmanager/0.log" Aug 04 13:28:40.495634 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.495603 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/config-reloader/0.log" Aug 04 13:28:40.524513 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.524483 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/kube-rbac-proxy-web/0.log" Aug 04 13:28:40.552393 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.552363 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/kube-rbac-proxy/0.log" Aug 04 13:28:40.583263 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.581884 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/kube-rbac-proxy-metric/0.log" Aug 04 13:28:40.619605 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.619248 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/prom-label-proxy/0.log" Aug 04 13:28:40.647869 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.647838 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54742e38-f7aa-41b1-8181-14dcb22d11bb/init-config-reloader/0.log" Aug 04 13:28:40.748827 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.748713 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-gfz2f_15c66317-9eda-412f-85e2-9415e04e829c/kube-state-metrics/0.log" Aug 04 13:28:40.770517 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.770490 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-gfz2f_15c66317-9eda-412f-85e2-9415e04e829c/kube-rbac-proxy-main/0.log" Aug 04 13:28:40.796237 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.796177 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-gfz2f_15c66317-9eda-412f-85e2-9415e04e829c/kube-rbac-proxy-self/0.log" Aug 04 13:28:40.830917 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.830885 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d76c6449f-jp7q7_f6191714-9ae7-42f7-acdf-529da820a389/metrics-server/0.log" Aug 04 13:28:40.866390 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.866256 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-2l5jz_fb3b212a-694c-4fd0-959c-ae6b3d3797f4/monitoring-plugin/0.log" Aug 04 13:28:40.904943 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.904862 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j7h7_462251bf-8039-4bfe-b33d-4fcf0a94e3f0/node-exporter/0.log" Aug 04 13:28:40.936700 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.936677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j7h7_462251bf-8039-4bfe-b33d-4fcf0a94e3f0/kube-rbac-proxy/0.log" Aug 04 13:28:40.966361 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:40.966331 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5j7h7_462251bf-8039-4bfe-b33d-4fcf0a94e3f0/init-textfile/0.log" Aug 04 13:28:41.216552 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.216492 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wj8b8_150af977-817a-4471-98fe-37818dce152a/kube-rbac-proxy-main/0.log" Aug 04 13:28:41.250379 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.250321 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wj8b8_150af977-817a-4471-98fe-37818dce152a/kube-rbac-proxy-self/0.log" Aug 04 13:28:41.277843 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.277759 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-wj8b8_150af977-817a-4471-98fe-37818dce152a/openshift-state-metrics/0.log" Aug 04 13:28:41.325976 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.325920 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/prometheus/0.log" Aug 04 13:28:41.356635 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.356519 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/config-reloader/0.log" Aug 04 13:28:41.378196 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.378106 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/thanos-sidecar/0.log" Aug 04 13:28:41.405728 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.405671 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/kube-rbac-proxy-web/0.log" Aug 04 13:28:41.430409 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.430386 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/kube-rbac-proxy/0.log" Aug 04 13:28:41.456782 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.456752 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/kube-rbac-proxy-thanos/0.log" Aug 04 13:28:41.482950 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.482918 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_07229bba-8c2b-493a-b3d2-61d63f6c336e/init-config-reloader/0.log" Aug 04 13:28:41.524060 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.524026 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-j4x4b_0be2614a-2d46-476a-9a62-e2ee371bcf99/prometheus-operator/0.log" Aug 04 13:28:41.548631 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.548597 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-j4x4b_0be2614a-2d46-476a-9a62-e2ee371bcf99/kube-rbac-proxy/0.log" Aug 04 13:28:41.739231 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.739134 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584c6647fc-tdbpc_f35733d2-ce07-4b1d-ab43-bd6e9b437271/thanos-query/0.log" Aug 04 13:28:41.805546 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.805493 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584c6647fc-tdbpc_f35733d2-ce07-4b1d-ab43-bd6e9b437271/kube-rbac-proxy-web/0.log" Aug 04 13:28:41.858419 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.858391 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584c6647fc-tdbpc_f35733d2-ce07-4b1d-ab43-bd6e9b437271/kube-rbac-proxy/0.log" Aug 04 13:28:41.921782 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.921752 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584c6647fc-tdbpc_f35733d2-ce07-4b1d-ab43-bd6e9b437271/prom-label-proxy/0.log" Aug 04 13:28:41.952268 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.952236 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584c6647fc-tdbpc_f35733d2-ce07-4b1d-ab43-bd6e9b437271/kube-rbac-proxy-rules/0.log" Aug 04 13:28:41.977701 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:41.977668 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-584c6647fc-tdbpc_f35733d2-ce07-4b1d-ab43-bd6e9b437271/kube-rbac-proxy-metrics/0.log" Aug 04 13:28:44.005232 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.005194 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f"] Aug 04 13:28:44.011246 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.011217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.014761 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.014732 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kqx2z\"/\"default-dockercfg-6pz8c\"" Aug 04 13:28:44.027398 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.027361 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f"] Aug 04 13:28:44.061842 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.061806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-lib-modules\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.062102 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.062079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-proc\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.062301 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.062285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-podres\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.062476 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.062462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-sys\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.062620 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.062602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8xd\" (UniqueName: \"kubernetes.io/projected/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-kube-api-access-mv8xd\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.163636 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.163585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-podres\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.164082 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.164064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-sys\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.164338 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.164322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8xd\" (UniqueName: \"kubernetes.io/projected/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-kube-api-access-mv8xd\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.164825 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.164804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-lib-modules\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.165086 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.165072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-proc\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.165251 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.164246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-sys\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.165412 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.165021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-lib-modules\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.165499 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.163947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-podres\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.165577 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.165370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-proc\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.176650 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.176628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8xd\" (UniqueName: \"kubernetes.io/projected/50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5-kube-api-access-mv8xd\") pod \"perf-node-gather-daemonset-nhn4f\" (UID: \"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5\") " pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.323671 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.323586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:44.511151 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.507894 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f"] Aug 04 13:28:44.693449 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:44.693418 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d9497cb75-fzgk8_d3529031-70d2-45fa-964e-1f0664f7edd6/console/0.log" Aug 04 13:28:45.500727 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:45.500680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" event={"ID":"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5","Type":"ContainerStarted","Data":"30bb3a7abcc85d69e32f97fbbb4aeded8ab41d8c8e54c817957423ad234392b3"} Aug 04 13:28:45.501264 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:45.500759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" event={"ID":"50c53b49-5ec9-4c2b-9f5f-bf1aacb270d5","Type":"ContainerStarted","Data":"ddac8686d199cf27b787e918f1b02fd04b10a211ed0df5a76c049376221e6403"} Aug 04 13:28:45.501666 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:45.501637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:46.569060 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:46.569025 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gwf96_537a9c5d-b2f4-4410-8a28-ff7fb59f5b96/dns/0.log" Aug 04 13:28:46.622719 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:46.622691 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gwf96_537a9c5d-b2f4-4410-8a28-ff7fb59f5b96/kube-rbac-proxy/0.log" Aug 04 13:28:46.788276 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:46.788249 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7vggh_8f3b799a-900a-4bff-be86-986adc410241/dns-node-resolver/0.log" Aug 04 13:28:47.585411 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:47.585382 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9dwhz_fc7df39c-e1e8-45a5-a852-a75592bd193e/node-ca/0.log" Aug 04 13:28:49.104081 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:49.104007 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688595d9d-hvkqz_a3bce0e2-a1cc-4759-a048-cf707c9ffe66/kube-auth-proxy/0.log" Aug 04 13:28:49.228138 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:49.228096 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wjhrj_34420b0a-98d8-4359-b89d-574811fe369d/payload-processing/0.log" Aug 04 13:28:50.154586 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:50.154555 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ncsgj_546cdf59-c034-4ab6-a386-f8ddfe14aeb3/serve-healthcheck-canary/0.log" Aug 04 13:28:50.877214 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:50.877178 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndt2d_ed9d7305-2faa-4e74-a7b6-f90c95f59092/kube-rbac-proxy/0.log" Aug 04 13:28:50.904782 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:50.904732 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndt2d_ed9d7305-2faa-4e74-a7b6-f90c95f59092/exporter/0.log" Aug 04 13:28:50.931531 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:50.931474 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndt2d_ed9d7305-2faa-4e74-a7b6-f90c95f59092/extractor/0.log" Aug 04 13:28:52.525175 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:52.525144 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" Aug 04 13:28:52.547172 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:52.547096 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kqx2z/perf-node-gather-daemonset-nhn4f" podStartSLOduration=9.547080312 podStartE2EDuration="9.547080312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 13:28:45.520284155 +0000 UTC m=+3656.475879952" watchObservedRunningTime="2026-08-04 13:28:52.547080312 +0000 UTC m=+3663.502676105" Aug 04 13:28:53.697987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:53.697454 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68d8768cdd-c7g2g_b6598625-e743-4b38-8ef5-7f62371e40e8/manager/0.log" Aug 04 13:28:54.098659 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:54.098631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547f69df5b-9lrrc_3654256c-6789-438c-97f0-08d6c2ef0be2/manager/0.log" Aug 04 13:28:54.129987 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:54.129948 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-547f69df5b-9lrrc_3654256c-6789-438c-97f0-08d6c2ef0be2/copy-manifests/0.log" Aug 04 13:28:55.172657 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:55.172561 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688595d9d-hvkqz_a3bce0e2-a1cc-4759-a048-cf707c9ffe66/kube-auth-proxy/0.log" Aug 04 13:28:55.269658 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:55.269629 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wjhrj_34420b0a-98d8-4359-b89d-574811fe369d/payload-processing/0.log" Aug 04 13:28:55.626010 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:55.625982 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c4cd7f74c-8dgn8_ed7cdc18-0e8e-47b0-86ab-1348fb415be5/manager/0.log" Aug 04 13:28:57.112380 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:57.112356 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5688595d9d-hvkqz_a3bce0e2-a1cc-4759-a048-cf707c9ffe66/kube-auth-proxy/0.log" Aug 04 13:28:57.192978 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:28:57.192945 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-wjhrj_34420b0a-98d8-4359-b89d-574811fe369d/payload-processing/0.log" Aug 04 13:29:02.583599 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.583569 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/kube-multus-additional-cni-plugins/0.log" Aug 04 13:29:02.611265 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.611231 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/egress-router-binary-copy/0.log" Aug 04 13:29:02.636330 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.636305 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/cni-plugins/0.log" Aug 04 13:29:02.660850 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.660823 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/bond-cni-plugin/0.log" Aug 04 13:29:02.686155 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.686106 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/routeoverride-cni/0.log" Aug 04 13:29:02.729971 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.729904 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/whereabouts-cni-bincopy/0.log" Aug 04 13:29:02.779311 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:02.779283 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tlnnv_564ce3c2-fbd1-400a-993d-7310abe7a910/whereabouts-cni/0.log" Aug 04 13:29:03.036426 ip-10-0-136-46 kubenswrapper[2578]: I0804 13:29:03.036352 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bc27s_cc89e5a3-43c5-4413-85ce-67c1df2eb916/kube-multus/0.log"